[2025-07-22 13:29:39,724][ INFO][PID:674034] Marking build as starting [2025-07-22 13:29:39,977][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:29:39,978][ INFO][PID:674034] VM allocation process starts [2025-07-22 13:29:40,001][ INFO][PID:674034] Trying to allocate VM: ResallocHost, ticket_id=4026483, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-07-22 13:29:45,003][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:29:50,004][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:29:55,005][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:00,006][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:05,007][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:10,008][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:15,012][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:20,013][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:25,014][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:30,015][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:35,016][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:40,018][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:45,025][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:50,026][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:30:55,027][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:00,028][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:05,031][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:10,032][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:15,034][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:20,042][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:25,045][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:30,052][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:35,054][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:40,056][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:45,058][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:50,059][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:31:55,060][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:00,062][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:05,064][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:10,066][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:15,067][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:20,069][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:25,070][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:30,071][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:35,073][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:40,075][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:45,086][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:50,091][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:32:55,092][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:00,094][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:05,097][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:10,101][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:15,103][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:20,104][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:25,105][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:30,106][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:35,108][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:40,110][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:45,115][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:50,116][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:55,117][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:33:58,147][ INFO][PID:674034] Allocated host ResallocHost, ticket_id=4026483, hostname=140.211.168.163, name=copr_osuosl_p09_prod_05724642_20250722_132938, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-07-22 13:33:58,148][ INFO][PID:674034] Allocating ssh connection to builder [2025-07-22 13:33:58,149][ INFO][PID:674034] Checking that builder machine is OK [2025-07-22 13:33:59,550][ INFO][PID:674034] Running remote command: copr-builder-ready epel-10-ppc64le [2025-07-22 13:34:00,619][ INFO][PID:674034] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-07-22 13:34:00,621][ INFO][PID:674034] Filling build.info file with builder info [2025-07-22 13:34:00,622][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:00,624][ INFO][PID:674034] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@OpenVPN", "project_name": "openvpn-release-2.6", "project_dirname": "openvpn-release-2.6", "submitter": "gitlab.com:flichtenheld", "ended_on": null, "started_on": 1753191240.6243503, "submitted_on": null, "status": 3, "chroot": "epel-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9297506-epel-10-ppc64le", "build_id": 9297506, "package_name": "ovpn-dco", "package_version": "0.2.20250718-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@OpenVPN/openvpn-release-2.6/ovpn-dco", "git_hash": "387512efd41bfa46a3b9c447e47c618ebfc24060", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@OpenVPN/openvpn-release-2.6--gitlab.com:flichtenheld", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-release-2.6", "result_dir": "09297506-ovpn-dco", "built_packages": "", "id": 9297506, "mockchain_macros": { "copr_username": "@OpenVPN", "copr_projectname": "openvpn-release-2.6", "vendor": "Fedora Project COPR (@OpenVPN/openvpn-release-2.6)" } } ] } [2025-07-22 13:34:00,667][ INFO][PID:674034] Sending fedora-messaging bus message in build.start [2025-07-22 13:34:01,822][ INFO][PID:674034] Sending fedora-messaging bus message in chroot.start [2025-07-22 13:34:01,855][ INFO][PID:674034] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9297506-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2025-07-22 13:34:03,181][ INFO][PID:674034] Downloading the builder-live.log file, attempt 1 [2025-07-22 13:34:03,187][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.163 copr-rpmbuild-log [2025-07-22 13:34:08,197][ INFO][PID:674034] Periodic builder liveness probe: alive [2025-07-22 13:34:08,199][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:13,201][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:18,203][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:23,205][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:28,208][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:33,209][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:38,211][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:43,212][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:48,213][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:53,214][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:34:58,215][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:03,217][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:08,218][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:13,220][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:18,221][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:23,223][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:28,226][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:33,228][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:38,233][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:43,243][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:48,244][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:53,246][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:35:58,248][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:03,250][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:08,251][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:13,253][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:18,254][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:23,255][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:28,256][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:33,258][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:38,259][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:43,261][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:48,262][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:53,263][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:36:58,265][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:03,268][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:08,269][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:13,271][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:18,275][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:23,277][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:25,139][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:37:28,278][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:33,279][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:35,140][WARNING][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 13:37:38,281][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:43,285][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:45,142][ INFO][PID:674034] Downloading the builder-live.log file, attempt 2 [2025-07-22 13:37:45,143][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.163 copr-rpmbuild-log [2025-07-22 13:37:48,286][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:53,287][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:37:58,288][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:03,289][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:08,291][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:13,292][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:18,293][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:23,294][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:28,296][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:33,297][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:38,299][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:43,301][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:45,282][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:38:48,303][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:53,304][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:38:55,286][WARNING][PID:674034] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 13:38:58,305][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:03,306][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:05,287][ INFO][PID:674034] Downloading the builder-live.log file, attempt 3 [2025-07-22 13:39:05,289][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.163 copr-rpmbuild-log [2025-07-22 13:39:08,308][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:13,309][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:18,311][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:23,313][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:28,316][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:33,317][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:38,323][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:43,326][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:48,328][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:53,329][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:39:58,331][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:03,333][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:05,426][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:40:08,335][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:13,336][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:15,429][WARNING][PID:674034] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 13:40:18,338][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:23,341][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:25,430][ INFO][PID:674034] Downloading the builder-live.log file, attempt 4 [2025-07-22 13:40:25,431][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.163 copr-rpmbuild-log [2025-07-22 13:40:28,342][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:33,343][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:38,345][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:43,346][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:48,347][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:53,349][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:40:58,350][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:03,351][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:08,353][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:13,354][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:18,355][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:23,356][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:25,572][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:41:28,358][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:33,359][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:35,576][ ERROR][PID:674034] Giving up for unstable SSH, failures: 1753191455.1404974, 1753191535.286077, 1753191615.429176, 1753191695.5766966 [2025-07-22 13:41:35,577][ ERROR][PID:674034] Re-try request for task on 'ResallocHost, ticket_id=4026483, hostname=140.211.168.163, name=copr_osuosl_p09_prod_05724642_20250722_132938, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-22 13:41:35,578][ INFO][PID:674034] Releasing VM back to pool [2025-07-22 13:41:35,586][ INFO][PID:674034] Retry #1 (on other host) [2025-07-22 13:41:35,587][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:35,588][ INFO][PID:674034] VM allocation process starts [2025-07-22 13:41:35,637][ INFO][PID:674034] Trying to allocate VM: ResallocHost, ticket_id=4026602, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-07-22 13:41:38,679][ INFO][PID:674034] Allocated host ResallocHost, ticket_id=4026602, hostname=140.211.168.5, name=copr_osuosl_p09_prod_05724813_20250722_133306, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-07-22 13:41:38,680][ INFO][PID:674034] Allocating ssh connection to builder [2025-07-22 13:41:38,680][ INFO][PID:674034] Checking that builder machine is OK [2025-07-22 13:41:40,144][ INFO][PID:674034] Running remote command: copr-builder-ready epel-10-ppc64le [2025-07-22 13:41:41,176][ INFO][PID:674034] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-07-22 13:41:41,177][ INFO][PID:674034] Filling build.info file with builder info [2025-07-22 13:41:41,180][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:41,181][ INFO][PID:674034] Not re-notifying FE and msg buses for the new host. [2025-07-22 13:41:41,181][ INFO][PID:674034] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9297506-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2025-07-22 13:41:42,604][ INFO][PID:674034] Downloading the builder-live.log file, attempt 1 [2025-07-22 13:41:42,605][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-07-22 13:41:47,604][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:52,605][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:41:57,606][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:02,607][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:07,608][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:12,609][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:17,610][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:22,612][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:27,613][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:32,614][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:37,615][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:42,616][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:47,618][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:52,619][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:42:57,620][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:02,622][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:07,623][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:12,624][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:17,625][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:22,627][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:27,628][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:32,629][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:37,630][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:42,631][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:47,632][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:52,633][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:43:57,634][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:02,635][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:07,637][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:12,642][ INFO][PID:674034] Periodic builder liveness probe: alive [2025-07-22 13:44:12,643][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:17,644][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:22,645][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:27,646][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:32,648][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:37,649][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:42,650][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:47,651][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:52,652][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:44:57,653][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:02,654][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:07,655][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:12,656][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:17,657][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:22,658][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:27,659][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:32,661][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:37,662][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:42,663][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:47,664][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:52,665][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:45:57,666][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:02,667][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:07,668][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:09,125][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:46:12,669][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:17,670][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:19,129][WARNING][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 13:46:22,672][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:27,673][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:29,129][ INFO][PID:674034] Downloading the builder-live.log file, attempt 2 [2025-07-22 13:46:29,131][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-07-22 13:46:32,674][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:37,675][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:42,676][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:47,677][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:52,678][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:46:57,679][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:02,680][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:07,682][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:12,683][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:17,684][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:22,685][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:27,691][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:29,266][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:47:32,693][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:37,694][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:39,271][WARNING][PID:674034] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 13:47:42,695][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:47,696][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:49,272][ INFO][PID:674034] Downloading the builder-live.log file, attempt 3 [2025-07-22 13:47:49,273][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-07-22 13:47:52,698][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:47:57,699][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:02,700][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:07,702][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:12,705][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:17,706][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:22,707][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:27,708][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:32,709][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:37,710][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:42,711][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:47,712][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:49,410][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:48:52,713][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:57,714][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:48:59,521][WARNING][PID:674034] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 13:49:02,715][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:07,716][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:09,521][ INFO][PID:674034] Downloading the builder-live.log file, attempt 4 [2025-07-22 13:49:09,523][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-07-22 13:49:12,717][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:17,718][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:22,720][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:27,721][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:32,722][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:37,724][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:42,725][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:47,726][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:52,727][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:49:57,728][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:02,729][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:07,731][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:09,662][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 13:50:12,732][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:17,733][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:19,674][ ERROR][PID:674034] Giving up for unstable SSH, failures: 1753191979.1291103, 1753192059.271351, 1753192139.521044, 1753192219.673966 [2025-07-22 13:50:19,675][ ERROR][PID:674034] Re-try request for task on 'ResallocHost, ticket_id=4026602, hostname=140.211.168.5, name=copr_osuosl_p09_prod_05724813_20250722_133306, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-22 13:50:19,675][ INFO][PID:674034] Releasing VM back to pool [2025-07-22 13:50:19,685][ INFO][PID:674034] Retry #2 (on other host) [2025-07-22 13:50:19,685][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:19,686][ INFO][PID:674034] VM allocation process starts [2025-07-22 13:50:19,742][ INFO][PID:674034] Trying to allocate VM: ResallocHost, ticket_id=4026674, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-07-22 13:50:22,782][ INFO][PID:674034] Allocated host ResallocHost, ticket_id=4026674, hostname=140.211.168.243, name=copr_osuosl_p09_prod_05724995_20250722_133552, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-07-22 13:50:22,783][ INFO][PID:674034] Allocating ssh connection to builder [2025-07-22 13:50:22,783][ INFO][PID:674034] Checking that builder machine is OK [2025-07-22 13:50:24,198][ INFO][PID:674034] Running remote command: copr-builder-ready epel-10-ppc64le [2025-07-22 13:50:25,222][ INFO][PID:674034] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-07-22 13:50:25,223][ INFO][PID:674034] Filling build.info file with builder info [2025-07-22 13:50:25,226][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:25,227][ INFO][PID:674034] Not re-notifying FE and msg buses for the new host. [2025-07-22 13:50:25,227][ INFO][PID:674034] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9297506-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2025-07-22 13:50:26,567][ INFO][PID:674034] Downloading the builder-live.log file, attempt 1 [2025-07-22 13:50:26,568][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log [2025-07-22 13:50:31,568][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:36,569][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:41,570][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:46,571][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:51,572][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:50:56,573][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:01,574][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:06,576][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:11,577][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:16,578][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:21,580][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:26,581][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:31,582][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:36,583][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:41,584][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:46,585][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:51,586][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:51:56,587][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:01,588][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:06,589][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:11,590][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:16,591][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:21,592][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:26,593][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:31,595][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:36,596][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:41,597][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:46,598][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:51,599][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:52:56,600][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:01,601][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:06,602][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:11,603][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:16,604][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:21,605][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:26,606][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:31,607][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:36,608][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:41,610][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:46,611][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:51,612][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:53:56,613][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:01,614][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:06,615][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:11,617][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:16,623][ INFO][PID:674034] Periodic builder liveness probe: alive [2025-07-22 13:54:16,623][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:21,624][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:26,625][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:31,632][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:36,637][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:41,646][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:46,650][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:51,651][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:54:56,652][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:01,654][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:06,656][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:11,657][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:16,658][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:21,660][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:26,670][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:31,677][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:36,679][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:41,680][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:46,681][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:51,682][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:55:56,683][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:01,684][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:06,685][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:11,686][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:16,688][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:21,691][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:26,693][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:31,703][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:36,708][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:41,718][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:46,719][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:51,720][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:56:56,725][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:01,733][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:06,743][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:11,746][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:16,752][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:21,755][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:26,758][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:31,761][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:36,771][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:41,778][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:46,780][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:51,786][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:57:56,797][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:01,800][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:06,807][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:11,812][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:16,814][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:21,826][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:26,833][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:31,839][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:36,844][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:41,852][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:46,857][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:51,862][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:58:56,864][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:01,865][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:06,870][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:11,877][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:16,879][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:21,884][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:26,891][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:31,894][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:36,897][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:41,899][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:46,903][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:51,904][ INFO][PID:674034] Checking for cancel request [2025-07-22 13:59:56,908][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:01,911][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:06,913][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:11,916][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:16,922][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:21,927][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:26,929][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:31,931][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:36,940][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:41,942][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:46,947][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:51,950][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:00:56,955][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:01,957][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:06,960][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:11,963][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:16,964][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:21,973][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:26,974][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:31,976][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:36,985][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:41,989][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:46,992][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:51,996][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:01:57,004][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:02,008][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:07,017][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:12,023][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:17,028][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:22,031][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:27,033][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:32,037][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:37,040][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:42,041][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:47,043][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:52,044][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:02:57,046][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:02,047][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:07,054][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:12,057][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:17,058][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:22,065][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:27,071][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:32,072][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:37,073][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:42,074][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:47,076][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:52,078][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:03:57,079][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:02,081][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:07,082][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:12,083][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:17,097][ INFO][PID:674034] Periodic builder liveness probe: alive [2025-07-22 14:04:17,098][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:22,099][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:27,105][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:32,108][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:37,110][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:42,114][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:47,115][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:52,117][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:04:57,121][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:02,129][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:07,132][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:12,134][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:17,144][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:22,152][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:27,159][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:32,165][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:37,171][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:42,172][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:47,174][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:52,175][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:05:57,179][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:02,181][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:07,183][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:12,187][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:17,192][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:22,193][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:27,195][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:32,197][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:37,199][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:42,203][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:47,207][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:52,208][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:06:57,215][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:02,218][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:07,220][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:12,224][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:17,227][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:22,229][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:27,238][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:32,241][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:37,246][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:42,247][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:47,250][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:52,272][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:07:57,280][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:02,289][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:07,297][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:12,298][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:17,308][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:22,312][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:27,315][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:32,332][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:37,334][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:42,379][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:47,381][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:52,386][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:08:57,396][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:02,408][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:07,412][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:12,420][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:17,425][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:22,431][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:27,435][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:32,442][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:37,451][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:42,454][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:47,459][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:52,463][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:09:57,476][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:02,479][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:07,480][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:12,483][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:17,487][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:22,493][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:27,495][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:32,497][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:37,502][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:42,503][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:47,504][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:52,507][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:10:57,514][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:02,516][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:07,519][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:12,524][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:17,529][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:22,530][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:27,533][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:32,535][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:37,549][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:42,550][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:47,558][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:52,560][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:11:57,564][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:02,569][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:07,574][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:12,577][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:17,582][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:22,584][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:27,586][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:32,590][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:37,592][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:42,594][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:47,596][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:52,599][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:12:57,602][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:02,606][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:07,608][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:12,614][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:17,618][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:22,620][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:27,628][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:32,632][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:37,635][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:42,641][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:47,646][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:52,648][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:13:57,651][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:14:02,653][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:14:07,662][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:14:12,663][ INFO][PID:674034] Checking for cancel request [2025-07-22 14:14:17,680][ INFO][PID:674034] Periodic builder liveness probe: dead [2025-07-22 14:15:17,839][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Connection timed out during banner exchange Connection to 140.211.168.243 port 22 timed out [2025-07-22 14:16:27,983][ ERROR][PID:674034] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: Connection timed out during banner exchange Connection to 140.211.168.243 port 22 timed out [2025-07-22 14:17:38,053][ ERROR][PID:674034] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.243 port 22: Connection timed out [2025-07-22 14:18:02,140][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 14:18:12,141][WARNING][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 14:18:22,142][ INFO][PID:674034] Downloading the builder-live.log file, attempt 2 [2025-07-22 14:18:22,143][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log [2025-07-22 14:18:48,123][ ERROR][PID:674034] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.243 port 22: Connection timed out [2025-07-22 14:18:58,124][ ERROR][PID:674034] Can't ssh to cancel build. [2025-07-22 14:19:22,212][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 14:19:32,215][WARNING][PID:674034] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 14:19:42,217][ INFO][PID:674034] Downloading the builder-live.log file, attempt 3 [2025-07-22 14:19:42,219][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log [2025-07-22 14:20:42,289][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 14:20:52,294][WARNING][PID:674034] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-22 14:21:02,295][ INFO][PID:674034] Downloading the builder-live.log file, attempt 4 [2025-07-22 14:21:02,296][ INFO][PID:674034] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log [2025-07-22 14:22:02,364][ ERROR][PID:674034] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-22 14:22:12,369][ ERROR][PID:674034] Giving up for unstable SSH, failures: 1753193892.141617, 1753193972.215779, 1753194052.294334, 1753194132.3692083 [2025-07-22 14:22:12,370][ ERROR][PID:674034] Re-try request for task on 'ResallocHost, ticket_id=4026674, hostname=140.211.168.243, name=copr_osuosl_p09_prod_05724995_20250722_133552, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-07-22 14:22:12,370][ INFO][PID:674034] Releasing VM back to pool [2025-07-22 14:22:12,379][ ERROR][PID:674034] Backend process error: Three host tried without success: {'140.211.168.243', '140.211.168.163', '140.211.168.5'} [2025-07-22 14:22:12,380][WARNING][PID:674034] Switching not-finished job state to 'failed' [2025-07-22 14:22:12,380][ INFO][PID:674034] Worker failed build, took 1907.1531224250793 [2025-07-22 14:22:12,381][ INFO][PID:674034] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@OpenVPN", "project_name": "openvpn-release-2.6", "project_dirname": "openvpn-release-2.6", "submitter": "gitlab.com:flichtenheld", "ended_on": 1753194132.3802006, "started_on": 1753192225.2270782, "submitted_on": null, "status": 0, "chroot": "epel-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9297506-epel-10-ppc64le", "build_id": 9297506, "package_name": "ovpn-dco", "package_version": "0.2.20250718-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@OpenVPN/openvpn-release-2.6/ovpn-dco", "git_hash": "387512efd41bfa46a3b9c447e47c618ebfc24060", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@OpenVPN/openvpn-release-2.6--gitlab.com:flichtenheld", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-release-2.6", "result_dir": "09297506-ovpn-dco", "built_packages": "", "id": 9297506, "mockchain_macros": { "copr_username": "@OpenVPN", "copr_projectname": "openvpn-release-2.6", "vendor": "Fedora Project COPR (@OpenVPN/openvpn-release-2.6)" } } ] } [2025-07-22 14:22:12,423][ INFO][PID:674034] Sending fedora-messaging bus message in build.end [2025-07-22 14:22:12,449][ INFO][PID:674034] Compressing /var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/09297506-ovpn-dco/builder-live.log by gzip [2025-07-22 14:22:12,451][ INFO][PID:674034] Running command 'gzip /var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/09297506-ovpn-dco/builder-live.log' as PID 1122913 [2025-07-22 14:22:12,452][ INFO][PID:674034] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/09297506-ovpn-dco/builder-live.log) [2025-07-22 14:22:12,453][ INFO][PID:674034] Compressing /var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/09297506-ovpn-dco/backend.log by gzip [2025-07-22 14:22:12,454][ INFO][PID:674034] Running command 'gzip /var/lib/copr/public_html/results/@OpenVPN/openvpn-release-2.6/epel-10-ppc64le/09297506-ovpn-dco/backend.log' as PID 1122914