[2025-12-18 17:22:40,929][ INFO][PID:2366565] Marking build as starting [2025-12-18 17:22:40,999][ INFO][PID:2366565] Checking that /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/repodata/repomd.xml exists [2025-12-18 17:22:41,094][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:22:41,095][ INFO][PID:2366565] VM allocation process starts [2025-12-18 17:22:41,108][ INFO][PID:2366565] Trying to allocate VM: ResallocHost, ticket_id=427319, requested_tags=['copr_builder', 'arch_x86_64'] [2025-12-18 17:22:44,151][ INFO][PID:2366565] Allocated host ResallocHost, ticket_id=427319, hostname=3.81.20.205, name=aws_x86_64_spot_prod_00684911_20251218_171727, requested_tags=['copr_builder', 'arch_x86_64'] [2025-12-18 17:22:44,151][ INFO][PID:2366565] Allocating ssh connection to builder [2025-12-18 17:22:44,152][ INFO][PID:2366565] Checking that builder machine is OK [2025-12-18 17:22:44,473][ INFO][PID:2366565] Running remote command: copr-builder-ready rhel-9-x86_64 [2025-12-18 17:22:44,658][ INFO][PID:2366565] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-12-18 17:22:44,659][ INFO][PID:2366565] Filling build.info file with builder info [2025-12-18 17:22:44,875][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:22:44,876][ INFO][PID:2366565] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@osbuild", "project_name": "image-builder", "project_dirname": "image-builder", "submitter": "packit", "ended_on": null, "started_on": 1766078564.8761668, "submitted_on": null, "status": 3, "chroot": "rhel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9931884-rhel-9-x86_64", "build_id": 9931884, "package_name": "image-builder", "package_version": "43-1.20251218164547359209.main.12.ge205501", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@osbuild/image-builder/image-builder", "git_hash": "0a493e403e189a155b9fe8e52d364886252549b4", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@osbuild/image-builder--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@osbuild/image-builder/rhel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@osbuild/image-builder", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@osbuild/image-builder", "result_dir": "09931884-image-builder", "built_packages": "", "id": 9931884, "mockchain_macros": { "copr_username": "@osbuild", "copr_projectname": "image-builder", "vendor": "Fedora Project COPR (@osbuild/image-builder)" } } ] } [2025-12-18 17:22:44,935][ INFO][PID:2366565] Sending fedora-messaging bus message in build.start [2025-12-18 17:22:45,474][ INFO][PID:2366565] Sending fedora-messaging bus message in chroot.start [2025-12-18 17:22:45,502][ INFO][PID:2366565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9931884-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached [2025-12-18 17:22:45,808][ INFO][PID:2366565] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.81.20.205' (ED25519) to the list of known hosts. [2025-12-18 17:22:45,809][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 1 [2025-12-18 17:22:45,810][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.81.20.205 copr-rpmbuild-log [2025-12-18 17:22:50,814][ INFO][PID:2366565] Periodic builder liveness probe: alive [2025-12-18 17:22:50,815][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:22:55,815][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:00,816][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:05,817][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:10,819][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:15,820][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:20,821][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:25,822][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:30,828][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:35,829][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:40,830][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:45,831][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:50,832][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:23:55,833][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:00,834][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:05,835][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:10,836][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:15,841][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:20,845][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:25,846][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:29,678][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:24:30,851][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:35,853][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:39,679][WARNING][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:24:40,854][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:45,855][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:49,680][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 2 [2025-12-18 17:24:49,809][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.81.20.205 copr-rpmbuild-log [2025-12-18 17:24:50,856][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:24:55,858][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:00,859][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:05,860][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:10,861][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:15,862][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:20,863][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:25,864][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:30,866][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:35,867][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:40,869][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:45,870][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:49,877][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:25:50,871][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:25:55,872][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:00,307][WARNING][PID:2366565] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:26:00,873][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:05,873][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:10,381][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 3 [2025-12-18 17:26:10,382][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.81.20.205 copr-rpmbuild-log [2025-12-18 17:26:10,874][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:15,876][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:20,877][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:25,878][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:30,879][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:35,880][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:40,881][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:45,882][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:50,883][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:26:55,884][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:00,885][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:05,886][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:10,450][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:27:10,887][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:15,888][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:20,461][WARNING][PID:2366565] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:27:20,889][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:25,890][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:30,464][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 4 [2025-12-18 17:27:30,466][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.81.20.205 copr-rpmbuild-log [2025-12-18 17:27:30,891][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:35,892][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:40,893][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:45,894][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:50,895][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:27:55,896][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:00,897][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:05,898][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:10,899][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:15,900][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:20,901][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:25,902][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:30,532][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:28:30,905][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:35,906][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:40,537][ ERROR][PID:2366565] Giving up for unstable SSH, failures: 1766078679.6795032, 1766078760.3074005, 1766078840.461531, 1766078920.5370758 [2025-12-18 17:28:40,538][ ERROR][PID:2366565] Re-try request for task on 'ResallocHost, ticket_id=427319, hostname=3.81.20.205, name=aws_x86_64_spot_prod_00684911_20251218_171727, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-12-18 17:28:40,538][ INFO][PID:2366565] Releasing VM back to pool [2025-12-18 17:28:40,549][ INFO][PID:2366565] Retry #1 (on other host) [2025-12-18 17:28:40,549][ INFO][PID:2366565] Checking that /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/repodata/repomd.xml exists [2025-12-18 17:28:40,550][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:40,550][ INFO][PID:2366565] VM allocation process starts [2025-12-18 17:28:40,561][ INFO][PID:2366565] Trying to allocate VM: ResallocHost, ticket_id=427408, requested_tags=['copr_builder', 'arch_x86_64'] [2025-12-18 17:28:43,573][ INFO][PID:2366565] Allocated host ResallocHost, ticket_id=427408, hostname=3.85.189.74, name=aws_x86_64_spot_prod_00685201_20251218_172521, requested_tags=['copr_builder', 'arch_x86_64'] [2025-12-18 17:28:43,574][ INFO][PID:2366565] Allocating ssh connection to builder [2025-12-18 17:28:43,574][ INFO][PID:2366565] Checking that builder machine is OK [2025-12-18 17:28:43,899][ INFO][PID:2366565] Running remote command: copr-builder-ready rhel-9-x86_64 [2025-12-18 17:28:44,092][ INFO][PID:2366565] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-12-18 17:28:44,093][ INFO][PID:2366565] Filling build.info file with builder info [2025-12-18 17:28:44,454][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:44,455][ INFO][PID:2366565] Not re-notifying FE and msg buses for the new host. [2025-12-18 17:28:44,455][ INFO][PID:2366565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9931884-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached [2025-12-18 17:28:44,761][ INFO][PID:2366565] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.85.189.74' (ED25519) to the list of known hosts. [2025-12-18 17:28:44,762][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 1 [2025-12-18 17:28:44,763][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.189.74 copr-rpmbuild-log [2025-12-18 17:28:49,762][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:54,763][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:28:55,419][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:28:59,764][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:04,765][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:05,419][WARNING][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:29:09,766][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:14,767][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:15,455][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 2 [2025-12-18 17:29:15,456][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.189.74 copr-rpmbuild-log [2025-12-18 17:29:19,768][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:24,769][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:29,770][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:34,771][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:39,772][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:44,773][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:49,774][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:54,775][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:29:59,776][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:04,777][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:09,778][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:14,779][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:15,524][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:30:19,780][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:24,782][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:25,540][WARNING][PID:2366565] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:30:29,783][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:34,783][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:35,540][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 3 [2025-12-18 17:30:35,541][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.189.74 copr-rpmbuild-log [2025-12-18 17:30:39,784][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:44,785][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:49,786][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:54,787][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:30:59,788][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:04,789][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:09,790][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:14,791][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:19,792][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:24,793][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:29,794][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:34,795][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:35,608][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:31:39,796][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:44,797][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:45,622][WARNING][PID:2366565] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:31:49,798][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:54,799][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:31:55,623][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 4 [2025-12-18 17:31:55,624][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.189.74 copr-rpmbuild-log [2025-12-18 17:31:59,800][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:04,801][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:09,802][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:14,803][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:19,804][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:24,805][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:29,806][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:34,807][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:39,808][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:44,809][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:49,810][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:54,833][ INFO][PID:2366565] Periodic builder liveness probe: alive [2025-12-18 17:32:54,844][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:32:55,689][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:32:59,845][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:04,853][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:05,690][ ERROR][PID:2366565] Giving up for unstable SSH, failures: 1766078945.4196901, 1766079025.5400298, 1766079105.6225753, 1766079185.6901891 [2025-12-18 17:33:05,691][ ERROR][PID:2366565] Re-try request for task on 'ResallocHost, ticket_id=427408, hostname=3.85.189.74, name=aws_x86_64_spot_prod_00685201_20251218_172521, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-12-18 17:33:05,691][ INFO][PID:2366565] Releasing VM back to pool [2025-12-18 17:33:05,702][ INFO][PID:2366565] Retry #2 (on other host) [2025-12-18 17:33:05,703][ INFO][PID:2366565] Checking that /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/repodata/repomd.xml exists [2025-12-18 17:33:05,703][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:05,705][ INFO][PID:2366565] VM allocation process starts [2025-12-18 17:33:05,719][ INFO][PID:2366565] Trying to allocate VM: ResallocHost, ticket_id=427469, requested_tags=['copr_builder', 'arch_x86_64'] [2025-12-18 17:33:08,735][ INFO][PID:2366565] Allocated host ResallocHost, ticket_id=427469, hostname=44.201.36.78, name=aws_x86_64_spot_prod_00685237_20251218_172657, requested_tags=['copr_builder', 'arch_x86_64'] [2025-12-18 17:33:08,735][ INFO][PID:2366565] Allocating ssh connection to builder [2025-12-18 17:33:08,736][ INFO][PID:2366565] Checking that builder machine is OK [2025-12-18 17:33:09,086][ INFO][PID:2366565] Running remote command: copr-builder-ready rhel-9-x86_64 [2025-12-18 17:33:09,282][ INFO][PID:2366565] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-12-18 17:33:09,282][ INFO][PID:2366565] Filling build.info file with builder info [2025-12-18 17:33:09,369][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:09,370][ INFO][PID:2366565] Not re-notifying FE and msg buses for the new host. [2025-12-18 17:33:09,370][ INFO][PID:2366565] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9931884-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached [2025-12-18 17:33:09,693][ INFO][PID:2366565] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.201.36.78' (ED25519) to the list of known hosts. [2025-12-18 17:33:09,696][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 1 [2025-12-18 17:33:09,700][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.36.78 copr-rpmbuild-log [2025-12-18 17:33:14,698][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:19,699][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:24,700][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:29,701][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:34,702][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:39,703][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:44,704][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:49,705][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:54,358][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:33:54,706][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:33:59,707][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:04,385][WARNING][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:34:04,708][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:09,709][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:14,385][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 2 [2025-12-18 17:34:14,387][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.36.78 copr-rpmbuild-log [2025-12-18 17:34:14,710][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:19,711][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:24,712][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:29,713][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:34,714][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:39,715][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:44,716][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:49,718][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:54,719][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:34:59,721][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:04,722][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:09,723][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:14,443][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:35:14,724][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:19,725][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:24,446][WARNING][PID:2366565] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:35:24,726][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:29,727][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:34,447][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 3 [2025-12-18 17:35:34,448][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.36.78 copr-rpmbuild-log [2025-12-18 17:35:34,728][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:39,729][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:44,730][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:49,731][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:54,732][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:35:59,733][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:04,734][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:09,735][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:14,736][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:19,738][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:24,739][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:29,740][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:34,516][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:36:34,741][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:39,742][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:44,531][WARNING][PID:2366565] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-12-18 17:36:44,743][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:49,749][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:54,532][ INFO][PID:2366565] Downloading the builder-live.log file, attempt 4 [2025-12-18 17:36:54,533][ INFO][PID:2366565] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.36.78 copr-rpmbuild-log [2025-12-18 17:36:54,751][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:36:59,752][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:04,753][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:09,754][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:14,755][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:19,756][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:24,757][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:29,758][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:34,762][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:39,763][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:44,765][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:49,766][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:54,600][ ERROR][PID:2366565] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-12-18 17:37:54,767][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:37:59,768][ INFO][PID:2366565] Checking for cancel request [2025-12-18 17:38:04,626][ ERROR][PID:2366565] Giving up for unstable SSH, failures: 1766079244.3851314, 1766079324.4463637, 1766079404.5312657, 1766079484.62621 [2025-12-18 17:38:04,627][ ERROR][PID:2366565] Re-try request for task on 'ResallocHost, ticket_id=427469, hostname=44.201.36.78, name=aws_x86_64_spot_prod_00685237_20251218_172657, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-12-18 17:38:04,627][ INFO][PID:2366565] Releasing VM back to pool [2025-12-18 17:38:04,639][ ERROR][PID:2366565] Backend process error: Three host tried without success: {'44.201.36.78', '3.85.189.74', '3.81.20.205'} [2025-12-18 17:38:04,640][WARNING][PID:2366565] Switching not-finished job state to 'failed' [2025-12-18 17:38:04,640][ INFO][PID:2366565] Worker failed build, took 295.27004051208496 [2025-12-18 17:38:04,641][ INFO][PID:2366565] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@osbuild", "project_name": "image-builder", "project_dirname": "image-builder", "submitter": "packit", "ended_on": 1766079484.64005, "started_on": 1766079189.3700094, "submitted_on": null, "status": 0, "chroot": "rhel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9931884-rhel-9-x86_64", "build_id": 9931884, "package_name": "image-builder", "package_version": "43-1.20251218164547359209.main.12.ge205501", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@osbuild/image-builder/image-builder", "git_hash": "0a493e403e189a155b9fe8e52d364886252549b4", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@osbuild/image-builder--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@osbuild/image-builder/rhel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@osbuild/image-builder", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@osbuild/image-builder", "result_dir": "09931884-image-builder", "built_packages": "", "id": 9931884, "mockchain_macros": { "copr_username": "@osbuild", "copr_projectname": "image-builder", "vendor": "Fedora Project COPR (@osbuild/image-builder)" } } ] } [2025-12-18 17:38:04,704][ INFO][PID:2366565] Sending fedora-messaging bus message in build.end [2025-12-18 17:38:04,750][ INFO][PID:2366565] Compressing /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/09931884-image-builder/builder-live.log by gzip [2025-12-18 17:38:04,751][ INFO][PID:2366565] Running command 'gzip /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/09931884-image-builder/builder-live.log' as PID 2436360 [2025-12-18 17:38:04,753][ INFO][PID:2366565] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/09931884-image-builder/builder-live.log) [2025-12-18 17:38:04,754][ INFO][PID:2366565] Compressing /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/09931884-image-builder/backend.log by gzip [2025-12-18 17:38:04,754][ INFO][PID:2366565] Running command 'gzip /var/lib/copr/public_html/results/@osbuild/image-builder/rhel-9-x86_64/09931884-image-builder/backend.log' as PID 2436361