[2025-07-14 18:25:07,451][ INFO][PID:1077584] Marking build as starting [2025-07-14 18:25:07,498][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:07,499][ INFO][PID:1077584] VM allocation process starts [2025-07-14 18:25:07,517][ INFO][PID:1077584] Trying to allocate VM: ResallocHost, ticket_id=3924814, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-14 18:25:10,562][ INFO][PID:1077584] Allocated host ResallocHost, ticket_id=3924814, hostname=174.129.57.160, name=aws_x86_64_spot_prod_05386237_20250714_182217, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-14 18:25:10,563][ INFO][PID:1077584] Allocating ssh connection to builder [2025-07-14 18:25:10,566][ INFO][PID:1077584] Checking that builder machine is OK [2025-07-14 18:25:10,995][ INFO][PID:1077584] Running remote command: copr-builder-ready fedora-rawhide-x86_64 [2025-07-14 18:25:11,202][ INFO][PID:1077584] Red Hat subscription not needed for fedora-rawhide-x86_64 Builder is ready to be used [2025-07-14 18:25:11,205][ INFO][PID:1077584] Filling build.info file with builder info [2025-07-14 18:25:11,207][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:11,211][ INFO][PID:1077584] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "krouma", "project_name": "kubernetes", "project_dirname": "kubernetes", "submitter": "krouma", "ended_on": null, "started_on": 1752517511.2114906, "submitted_on": null, "status": 3, "chroot": "fedora-rawhide-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9272961-fedora-rawhide-x86_64", "build_id": 9272961, "package_name": "chectl", "package_version": "7.106.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/krouma/kubernetes/chectl", "git_hash": "fab6ffcb93efc5f310bec04a68f9755a4930d4df", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "krouma/kubernetes--krouma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/krouma/kubernetes/fedora-rawhide-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": true, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/krouma/kubernetes", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/krouma/kubernetes", "result_dir": "09272961-chectl", "built_packages": "", "id": 9272961, "mockchain_macros": { "copr_username": "krouma", "copr_projectname": "kubernetes", "vendor": "Fedora Project COPR (krouma/kubernetes)" } } ] } [2025-07-14 18:25:11,283][ INFO][PID:1077584] Sending fedora-messaging bus message in build.start [2025-07-14 18:25:12,386][ INFO][PID:1077584] Sending fedora-messaging bus message in chroot.start [2025-07-14 18:25:12,418][ INFO][PID:1077584] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9272961-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached [2025-07-14 18:25:12,764][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 1 [2025-07-14 18:25:12,766][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.57.160 copr-rpmbuild-log [2025-07-14 18:25:17,771][ INFO][PID:1077584] Periodic builder liveness probe: alive [2025-07-14 18:25:17,772][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:22,773][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:27,774][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:32,776][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:37,777][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:42,778][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:47,779][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:52,781][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:25:57,782][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:02,784][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:07,785][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:12,787][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:17,794][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:22,795][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:27,796][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:32,800][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:37,803][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:42,804][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:47,806][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:52,808][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:26:57,809][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:02,810][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:07,812][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:12,813][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:17,814][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:22,815][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:27,816][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:32,817][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:37,818][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:42,819][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:47,821][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:52,822][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:27:57,823][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:02,824][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:07,825][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:12,827][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:17,828][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:22,829][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:27,831][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:32,832][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:37,834][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:42,835][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:47,836][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:52,837][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:57,838][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:28:59,553][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:29:02,840][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:07,849][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:09,554][WARNING][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-14 18:29:12,850][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:17,851][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:19,555][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 2 [2025-07-14 18:29:19,557][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.57.160 copr-rpmbuild-log [2025-07-14 18:29:22,852][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:27,854][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:32,861][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:37,862][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:42,864][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:47,865][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:52,866][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:29:57,867][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:02,868][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:07,870][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:12,871][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:17,872][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:19,626][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:30:22,873][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:27,875][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:29,674][WARNING][PID:1077584] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-14 18:30:32,876][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:37,879][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:39,675][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 3 [2025-07-14 18:30:39,676][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.57.160 copr-rpmbuild-log [2025-07-14 18:30:42,880][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:47,882][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:52,883][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:30:57,884][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:02,886][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:07,887][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:12,888][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:17,890][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:22,891][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:27,892][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:32,894][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:37,898][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:39,746][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:31:42,901][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:47,904][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:49,805][WARNING][PID:1077584] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-14 18:31:52,906][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:57,909][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:31:59,808][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 4 [2025-07-14 18:31:59,810][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@174.129.57.160 copr-rpmbuild-log [2025-07-14 18:32:02,911][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:07,912][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:12,916][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:17,917][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:22,919][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:27,920][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:32,926][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:37,928][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:42,929][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:47,931][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:52,932][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:57,933][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:32:59,886][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:33:02,935][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:07,937][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:09,978][ ERROR][PID:1077584] Giving up for unstable SSH, failures: 1752517749.554274, 1752517829.6743658, 1752517909.8058643, 1752517989.978685 [2025-07-14 18:33:09,983][ ERROR][PID:1077584] Re-try request for task on 'ResallocHost, ticket_id=3924814, hostname=174.129.57.160, name=aws_x86_64_spot_prod_05386237_20250714_182217, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-14 18:33:09,984][ INFO][PID:1077584] Releasing VM back to pool [2025-07-14 18:33:09,998][ INFO][PID:1077584] Retry #1 (on other host) [2025-07-14 18:33:09,999][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:10,000][ INFO][PID:1077584] VM allocation process starts [2025-07-14 18:33:10,022][ INFO][PID:1077584] Trying to allocate VM: ResallocHost, ticket_id=3924943, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-14 18:33:13,063][ INFO][PID:1077584] Allocated host ResallocHost, ticket_id=3924943, hostname=54.91.228.133, name=aws_x86_64_spot_prod_05386190_20250714_182035, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-14 18:33:13,066][ INFO][PID:1077584] Allocating ssh connection to builder [2025-07-14 18:33:13,067][ INFO][PID:1077584] Checking that builder machine is OK [2025-07-14 18:33:13,617][ INFO][PID:1077584] Running remote command: copr-builder-ready fedora-rawhide-x86_64 [2025-07-14 18:33:13,844][ INFO][PID:1077584] Red Hat subscription not needed for fedora-rawhide-x86_64 Builder is ready to be used [2025-07-14 18:33:13,847][ INFO][PID:1077584] Filling build.info file with builder info [2025-07-14 18:33:13,853][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:13,857][ INFO][PID:1077584] Not re-notifying FE and msg buses for the new host. [2025-07-14 18:33:13,859][ INFO][PID:1077584] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9272961-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached [2025-07-14 18:33:14,234][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 1 [2025-07-14 18:33:14,238][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.91.228.133 copr-rpmbuild-log [2025-07-14 18:33:19,235][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:24,236][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:29,241][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:34,244][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:39,248][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:44,249][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:49,251][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:54,258][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:33:59,263][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:04,272][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:08,674][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:34:09,274][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:14,279][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:18,675][WARNING][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-14 18:34:19,281][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:24,288][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:28,676][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 2 [2025-07-14 18:34:28,678][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.91.228.133 copr-rpmbuild-log [2025-07-14 18:34:29,289][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:34,291][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:39,292][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:44,296][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:49,297][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:54,303][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:34:59,304][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:04,305][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:09,306][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:14,309][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:19,331][ INFO][PID:1077584] Periodic builder liveness probe: alive [2025-07-14 18:35:19,332][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:24,334][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:28,746][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:35:29,336][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:34,337][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:38,926][WARNING][PID:1077584] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-14 18:35:39,339][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:44,342][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:48,927][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 3 [2025-07-14 18:35:48,930][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.91.228.133 copr-rpmbuild-log [2025-07-14 18:35:49,343][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:54,345][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:35:59,346][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:04,348][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:09,350][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:14,351][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:19,353][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:24,357][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:29,358][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:34,360][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:39,361][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:44,364][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:48,999][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:36:49,365][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:54,366][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:36:59,152][WARNING][PID:1077584] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-14 18:36:59,368][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:04,369][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:09,153][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 4 [2025-07-14 18:37:09,154][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.91.228.133 copr-rpmbuild-log [2025-07-14 18:37:09,370][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:14,371][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:19,372][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:24,373][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:29,375][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:34,376][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:39,377][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:44,378][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:49,380][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:54,381][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:37:59,382][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:04,383][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:09,223][ ERROR][PID:1077584] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-14 18:38:09,384][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:14,386][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:19,226][ ERROR][PID:1077584] Giving up for unstable SSH, failures: 1752518058.6753953, 1752518138.9265964, 1752518219.1522381, 1752518299.226173 [2025-07-14 18:38:19,228][ ERROR][PID:1077584] Re-try request for task on 'ResallocHost, ticket_id=3924943, hostname=54.91.228.133, name=aws_x86_64_spot_prod_05386190_20250714_182035, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-14 18:38:19,229][ INFO][PID:1077584] Releasing VM back to pool [2025-07-14 18:38:19,238][ INFO][PID:1077584] Retry #2 (on other host) [2025-07-14 18:38:19,239][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:19,240][ INFO][PID:1077584] VM allocation process starts [2025-07-14 18:38:19,256][ INFO][PID:1077584] Trying to allocate VM: ResallocHost, ticket_id=3925017, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-14 18:38:22,276][ INFO][PID:1077584] Allocated host ResallocHost, ticket_id=3925017, hostname=44.200.199.151, name=aws_x86_64_normalreserved_prod_05386238_20250714_182219, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-14 18:38:22,277][ INFO][PID:1077584] Allocating ssh connection to builder [2025-07-14 18:38:22,277][ INFO][PID:1077584] Checking that builder machine is OK [2025-07-14 18:38:22,733][ INFO][PID:1077584] Running remote command: copr-builder-ready fedora-rawhide-x86_64 [2025-07-14 18:38:23,064][ INFO][PID:1077584] Red Hat subscription not needed for fedora-rawhide-x86_64 Builder is ready to be used [2025-07-14 18:38:23,064][ INFO][PID:1077584] Filling build.info file with builder info [2025-07-14 18:38:23,082][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:23,084][ INFO][PID:1077584] Not re-notifying FE and msg buses for the new host. [2025-07-14 18:38:23,089][ INFO][PID:1077584] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9272961-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached [2025-07-14 18:38:23,573][ INFO][PID:1077584] Downloading the builder-live.log file, attempt 1 [2025-07-14 18:38:23,578][ INFO][PID:1077584] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.199.151 copr-rpmbuild-log [2025-07-14 18:38:28,573][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:33,575][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:38,577][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:43,578][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:48,579][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:53,582][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:38:58,584][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:03,585][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:08,586][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:13,587][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:18,588][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:23,591][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:28,592][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:33,597][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:38,602][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:43,607][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:48,609][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:53,612][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:39:58,613][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:03,616][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:08,621][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:13,626][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:18,630][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:23,633][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:28,635][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:33,638][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:38,640][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:43,643][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:48,646][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:53,649][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:40:58,655][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:03,657][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:08,658][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:13,661][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:18,663][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:23,666][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:28,668][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:33,669][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:38,680][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:43,684][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:48,687][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:53,693][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:41:58,695][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:03,697][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:08,701][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:13,702][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:18,703][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:23,706][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:28,707][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:33,711][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:38,713][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:43,720][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:48,724][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:53,731][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:42:58,733][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:03,737][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:08,743][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:13,745][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:18,749][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:23,751][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:28,755][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:33,787][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:38,790][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:43,792][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:48,793][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:53,795][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:43:58,799][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:03,805][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:08,808][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:13,809][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:18,815][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:23,817][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:28,827][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:33,828][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:38,832][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:43,846][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:48,858][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:53,882][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:44:58,887][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:03,909][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:08,915][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:14,009][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:19,018][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:24,081][ INFO][PID:1077584] Periodic builder liveness probe: alive [2025-07-14 18:45:24,081][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:29,086][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:34,088][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:39,104][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:44,111][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:49,120][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:54,125][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:45:59,132][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:04,133][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:09,134][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:14,137][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:19,139][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:24,155][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:29,163][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:34,167][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:39,168][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:44,171][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:49,178][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:54,199][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:46:59,200][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:04,202][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:09,206][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:14,210][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:19,213][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:24,215][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:29,217][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:34,230][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:39,234][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:44,278][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:49,279][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:54,280][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:47:59,283][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:04,284][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:09,286][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:14,323][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:19,336][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:24,348][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:29,360][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:34,362][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:39,368][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:44,369][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:49,370][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:54,371][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:48:59,431][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:04,434][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:09,464][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:14,466][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:19,584][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:24,586][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:29,588][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:34,589][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:39,592][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:44,594][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:49,596][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:54,604][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:49:59,608][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:04,609][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:09,612][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:14,675][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:19,677][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:24,679][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:29,681][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:34,683][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:39,692][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:44,704][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:49,705][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:54,706][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:50:59,807][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:51:04,809][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:51:09,811][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:51:14,813][ INFO][PID:1077584] Checking for cancel request [2025-07-14 18:51:16,595][ INFO][PID:1077584] Downloading results from builder [2025-07-14 18:51:16,596][ INFO][PID:1077584] rsyncing of mockbuilder@44.200.199.151:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl started [2025-07-14 18:51:16,597][ INFO][PID:1077584] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.200.199.151:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/ &> /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/build-09272961.rsync.log [2025-07-14 18:51:17,294][ INFO][PID:1077584] rsyncing finished. [2025-07-14 18:51:17,295][ INFO][PID:1077584] Releasing VM back to pool [2025-07-14 18:51:17,306][ INFO][PID:1077584] Searching for 'success' file in resultdir [2025-07-14 18:51:17,307][ INFO][PID:1077584] Going to sign pkgs from source: 9272961-fedora-rawhide-x86_64 in chroot: /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64 [2025-07-14 18:51:17,308][ INFO][PID:1077584] Calling '/bin/sign -u krouma#kubernetes@copr.fedorahosted.org -p' (attempt #1) [2025-07-14 18:51:17,646][ INFO][PID:1077584] Calling '/bin/sign -4 -h sha256 -u krouma#kubernetes@copr.fedorahosted.org -r /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/chectl-7.106.0-1.fc43.src.rpm' (attempt #1) [2025-07-14 18:51:18,390][ INFO][PID:1077584] signed rpm: /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/chectl-7.106.0-1.fc43.src.rpm [2025-07-14 18:51:18,391][ INFO][PID:1077584] Calling '/bin/sign -4 -h sha256 -u krouma#kubernetes@copr.fedorahosted.org -r /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/chectl-7.106.0-1.fc43.x86_64.rpm' (attempt #1) [2025-07-14 18:51:20,325][ INFO][PID:1077584] signed rpm: /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/chectl-7.106.0-1.fc43.x86_64.rpm [2025-07-14 18:51:20,326][ INFO][PID:1077584] Sign done [2025-07-14 18:51:20,327][ INFO][PID:1077584] Incremental createrepo run, adding 09272961-chectl into https://download.copr.fedorainfracloud.org/results/krouma/kubernetes/fedora-rawhide-x86_64, (auto-create-repo=True) [2025-07-14 18:51:20,328][ INFO][PID:1077584] Running command 'copr-repo --batched /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64 --add 09272961-chectl --no-appstream-metadata' as PID 1265936 [2025-07-14 18:51:21,716][ INFO][PID:1077584] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64 --add 09272961-chectl --no-appstream-metadata) [2025-07-14 18:51:21,718][ INFO][PID:1077584] Getting build details [2025-07-14 18:51:21,719][ INFO][PID:1077584] Listing built binary packages in /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl [2025-07-14 18:51:21,719][ INFO][PID:1077584] Built packages: ['chectl 7.106.0'] [2025-07-14 18:51:21,720][ INFO][PID:1077584] build details: {'built_packages': 'chectl 7.106.0'} [2025-07-14 18:51:21,720][ INFO][PID:1077584] Retrieving pubkey [2025-07-14 18:51:21,721][ INFO][PID:1077584] Calling '/bin/sign -u krouma#kubernetes@copr.fedorahosted.org -p' (attempt #1) [2025-07-14 18:51:22,059][ INFO][PID:1077584] Added pubkey for user krouma project kubernetes into: /var/lib/copr/public_html/results/krouma/kubernetes/pubkey.gpg [2025-07-14 18:51:22,059][ INFO][PID:1077584] Finished build: id=9272961 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/krouma/kubernetes chroot=fedora-rawhide-x86_64 [2025-07-14 18:51:22,062][ INFO][PID:1077584] Worker succeeded build, took 778.977034330368 [2025-07-14 18:51:22,063][ INFO][PID:1077584] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "krouma", "project_name": "kubernetes", "project_dirname": "kubernetes", "submitter": "krouma", "ended_on": 1752519082.0619755, "started_on": 1752518303.0849411, "submitted_on": null, "status": 1, "chroot": "fedora-rawhide-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9272961-fedora-rawhide-x86_64", "build_id": 9272961, "package_name": "chectl", "package_version": "7.106.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/krouma/kubernetes/chectl", "git_hash": "fab6ffcb93efc5f310bec04a68f9755a4930d4df", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "krouma/kubernetes--krouma", "results": { "packages": [ { "name": "chectl", "epoch": null, "version": "7.106.0", "release": "1.fc43", "arch": "src" }, { "name": "chectl", "epoch": null, "version": "7.106.0", "release": "1.fc43", "arch": "x86_64" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/krouma/kubernetes/fedora-rawhide-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": true, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/krouma/kubernetes", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/krouma/kubernetes", "result_dir": "09272961-chectl", "built_packages": "chectl 7.106.0", "id": 9272961, "mockchain_macros": { "copr_username": "krouma", "copr_projectname": "kubernetes", "vendor": "Fedora Project COPR (krouma/kubernetes)" } } ] } [2025-07-14 18:51:22,095][ INFO][PID:1077584] Sending fedora-messaging bus message in build.end [2025-07-14 18:51:22,145][ INFO][PID:1077584] Compressing /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/builder-live.log by gzip [2025-07-14 18:51:22,147][ INFO][PID:1077584] Running command 'gzip /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/builder-live.log' as PID 1266175 [2025-07-14 18:51:22,157][ INFO][PID:1077584] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/builder-live.log) [2025-07-14 18:51:22,157][ INFO][PID:1077584] Compressing /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/backend.log by gzip [2025-07-14 18:51:22,158][ INFO][PID:1077584] Running command 'gzip /var/lib/copr/public_html/results/krouma/kubernetes/fedora-rawhide-x86_64/09272961-chectl/backend.log' as PID 1266176