[2025-07-10 12:28:31,232][ INFO][PID:2181572] Marking build as starting [2025-07-10 12:28:31,420][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:28:31,421][ INFO][PID:2181572] VM allocation process starts [2025-07-10 12:28:31,444][ INFO][PID:2181572] Trying to allocate VM: ResallocHost, ticket_id=3878740, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-10 12:28:34,473][ INFO][PID:2181572] Allocated host ResallocHost, ticket_id=3878740, hostname=2620:52:3:1:dead:beef:cafe:c28f, name=copr_p09_01_prod_05248236_20250710_120601, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-10 12:28:34,474][ INFO][PID:2181572] Allocating ssh connection to builder [2025-07-10 12:28:34,474][ INFO][PID:2181572] Checking that builder machine is OK [2025-07-10 12:28:35,199][ INFO][PID:2181572] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-07-10 12:28:35,629][ INFO][PID:2181572] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-07-10 12:28:35,629][ INFO][PID:2181572] Filling build.info file with builder info [2025-07-10 12:28:35,631][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:28:35,632][ INFO][PID:2181572] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "nmstate", "project_name": "nmstate-git", "project_dirname": "nmstate-git", "submitter": "https://api.github.com/users/cathay4t", "ended_on": null, "started_on": 1752150515.6317785, "submitted_on": null, "status": 3, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9262832-fedora-41-ppc64le", "build_id": 9262832, "package_name": "nmstate", "package_version": "2.2.48-0.20250710.2581git13e44aa8", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/nmstate/nmstate-git/nmstate", "git_hash": "80378efcd5de8840356f3d64b82f378ad7f29b11", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "nmstate/nmstate-git--https://api.github.com/users/cathay4t", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/nmstate/nmstate-git/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/nmstate/nmstate-git", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/nmstate/nmstate-git", "result_dir": "09262832-nmstate", "built_packages": "", "id": 9262832, "mockchain_macros": { "copr_username": "nmstate", "copr_projectname": "nmstate-git", "vendor": "Fedora Project COPR (nmstate/nmstate-git)" } } ] } [2025-07-10 12:28:35,716][ INFO][PID:2181572] Sending fedora-messaging bus message in build.start [2025-07-10 12:28:36,520][ INFO][PID:2181572] Sending fedora-messaging bus message in chroot.start [2025-07-10 12:28:36,546][ INFO][PID:2181572] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9262832-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-07-10 12:28:37,169][ INFO][PID:2181572] Downloading the builder-live.log file, attempt 1 [2025-07-10 12:28:37,170][ INFO][PID:2181572] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28f copr-rpmbuild-log [2025-07-10 12:28:42,175][ INFO][PID:2181572] Periodic builder liveness probe: alive [2025-07-10 12:28:42,175][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:28:47,176][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:28:52,177][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:28:57,179][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:02,180][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:07,181][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:12,184][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:17,185][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:22,186][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:27,187][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:32,188][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:37,189][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:42,190][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:47,191][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:52,192][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:29:57,193][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:02,195][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:07,196][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:12,197][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:17,198][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:22,199][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:27,200][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:32,201][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:37,202][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:42,203][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:47,204][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:52,205][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:30:57,207][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:02,207][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:07,208][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:12,210][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:17,211][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:22,212][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:27,213][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:32,214][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:37,215][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:42,216][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:47,217][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:52,218][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:31:57,219][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:02,221][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:07,222][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:12,223][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:17,224][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:22,225][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:27,226][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:32,227][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:37,228][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:42,229][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:47,230][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:52,231][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:32:57,233][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:02,234][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:07,235][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:12,236][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:17,237][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:22,238][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:27,239][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:32,240][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:37,241][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:42,242][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:47,243][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:52,244][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:33:57,245][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:02,246][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:07,247][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:12,248][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:17,249][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:22,250][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:27,251][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:32,252][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:37,254][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:42,255][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:47,256][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:52,257][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:34:57,258][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:02,259][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:07,260][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:12,261][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:17,262][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:22,263][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:27,264][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:32,265][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:37,266][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:42,267][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:47,268][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:52,269][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:35:57,270][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:02,271][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:07,272][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:12,274][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:17,275][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:22,275][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:27,276][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:32,277][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:37,284][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:42,285][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:47,286][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:52,287][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:36:57,288][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:02,289][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:07,290][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:12,291][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:17,292][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:22,293][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:27,294][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:32,295][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:37,296][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:42,297][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:47,298][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:52,299][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:37:57,300][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:02,301][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:07,302][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:12,303][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:17,304][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:22,305][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:27,308][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:32,309][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:37,310][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:42,318][ INFO][PID:2181572] Periodic builder liveness probe: alive [2025-07-10 12:38:42,319][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:47,320][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:52,321][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:38:57,322][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:02,323][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:07,324][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:12,325][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:17,326][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:22,327][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:27,328][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:32,329][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:37,330][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:39,129][ ERROR][PID:2181572] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-10 12:39:42,331][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:47,332][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:49,129][WARNING][PID:2181572] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-10 12:39:52,333][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:57,334][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:39:59,130][ INFO][PID:2181572] Downloading the builder-live.log file, attempt 2 [2025-07-10 12:39:59,131][ INFO][PID:2181572] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28f copr-rpmbuild-log [2025-07-10 12:40:02,336][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:07,337][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:12,338][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:17,339][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:22,340][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:27,341][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:32,342][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:37,343][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:42,344][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:47,345][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:52,346][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:57,347][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:40:59,199][ ERROR][PID:2181572] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-10 12:41:02,348][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:07,350][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:09,201][WARNING][PID:2181572] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-10 12:41:12,351][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:17,352][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:19,202][ INFO][PID:2181572] Downloading the builder-live.log file, attempt 3 [2025-07-10 12:41:19,203][ INFO][PID:2181572] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28f copr-rpmbuild-log [2025-07-10 12:41:22,353][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:27,354][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:32,355][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:37,356][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:42,357][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:47,358][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:52,360][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:41:57,361][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:02,362][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:07,363][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:12,364][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:17,365][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:19,271][ ERROR][PID:2181572] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-10 12:42:22,366][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:27,367][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:29,272][WARNING][PID:2181572] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-10 12:42:32,368][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:37,369][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:39,272][ INFO][PID:2181572] Downloading the builder-live.log file, attempt 4 [2025-07-10 12:42:39,273][ INFO][PID:2181572] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28f copr-rpmbuild-log [2025-07-10 12:42:42,370][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:47,371][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:52,372][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:42:57,373][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:02,374][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:07,375][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:12,376][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:17,377][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:22,378][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:27,379][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:32,380][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:37,381][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:39,341][ ERROR][PID:2181572] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-10 12:43:42,382][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:47,383][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:49,342][ ERROR][PID:2181572] Giving up for unstable SSH, failures: 1752151189.129768, 1752151269.2017372, 1752151349.2720573, 1752151429.3424103 [2025-07-10 12:43:49,343][ ERROR][PID:2181572] Re-try request for task on 'ResallocHost, ticket_id=3878740, hostname=2620:52:3:1:dead:beef:cafe:c28f, name=copr_p09_01_prod_05248236_20250710_120601, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-10 12:43:49,343][ INFO][PID:2181572] Releasing VM back to pool [2025-07-10 12:43:49,352][ INFO][PID:2181572] Retry #1 (on other host) [2025-07-10 12:43:49,352][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:49,353][ INFO][PID:2181572] VM allocation process starts [2025-07-10 12:43:49,392][ INFO][PID:2181572] Trying to allocate VM: ResallocHost, ticket_id=3878817, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-10 12:43:52,437][ INFO][PID:2181572] Allocated host ResallocHost, ticket_id=3878817, hostname=2620:52:3:1:dead:beef:cafe:c2a0, name=copr_p09_01_prod_05248238_20250710_120602, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-10 12:43:52,437][ INFO][PID:2181572] Allocating ssh connection to builder [2025-07-10 12:43:52,438][ INFO][PID:2181572] Checking that builder machine is OK [2025-07-10 12:43:53,216][ INFO][PID:2181572] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-07-10 12:43:53,645][ INFO][PID:2181572] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-07-10 12:43:53,646][ INFO][PID:2181572] Filling build.info file with builder info [2025-07-10 12:43:53,646][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:43:53,647][ INFO][PID:2181572] Not re-notifying FE and msg buses for the new host. [2025-07-10 12:43:53,647][ INFO][PID:2181572] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9262832-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-07-10 12:43:54,337][ INFO][PID:2181572] Downloading the builder-live.log file, attempt 1 [2025-07-10 12:43:54,339][ INFO][PID:2181572] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a0 copr-rpmbuild-log [2025-07-10 12:43:59,338][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:04,339][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:09,340][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:14,341][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:19,341][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:24,342][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:29,343][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:34,344][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:39,345][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:44,347][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:49,347][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:54,349][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:44:59,349][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:04,350][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:09,351][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:14,352][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:19,353][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:24,355][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:29,356][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:34,357][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:39,358][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:44,359][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:49,360][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:54,361][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:45:59,362][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:04,363][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:09,364][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:14,365][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:19,366][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:24,367][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:29,368][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:34,369][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:39,370][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:44,371][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:49,372][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:54,373][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:46:59,374][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:04,375][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:09,376][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:14,377][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:19,378][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:24,379][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:29,380][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:34,381][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:39,382][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:44,383][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:49,384][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:54,385][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:47:59,386][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:04,387][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:09,388][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:14,389][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:19,390][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:24,391][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:29,392][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:34,395][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:39,399][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:44,412][ INFO][PID:2181572] Periodic builder liveness probe: alive [2025-07-10 12:48:44,413][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:49,414][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:54,415][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:48:59,417][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:04,418][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:09,419][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:14,420][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:19,421][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:24,422][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:29,423][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:34,424][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:39,425][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:44,426][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:49,427][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:54,429][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:49:59,430][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:04,431][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:09,432][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:14,433][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:19,438][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:24,439][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:29,441][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:34,442][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:39,443][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:44,444][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:49,445][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:54,447][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:50:59,448][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:04,449][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:09,451][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:14,454][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:19,456][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:24,458][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:29,465][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:34,466][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:39,467][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:44,469][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:49,472][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:54,473][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:51:59,481][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:04,484][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:09,486][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:14,487][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:19,489][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:24,491][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:29,492][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:34,493][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:39,494][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:44,495][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:49,497][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:54,498][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:52:59,499][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:04,501][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:09,502][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:14,504][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:19,505][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:24,507][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:29,508][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:34,509][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:39,510][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:44,511][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:49,512][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:54,513][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:53:59,514][ INFO][PID:2181572] Checking for cancel request [2025-07-10 12:54:02,780][ INFO][PID:2181572] Downloading results from builder [2025-07-10 12:54:02,780][ INFO][PID:2181572] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a0]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate started [2025-07-10 12:54:02,781][ INFO][PID:2181572] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a0]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/ &> /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/build-09262832.rsync.log [2025-07-10 12:54:04,043][ INFO][PID:2181572] rsyncing finished. [2025-07-10 12:54:04,043][ INFO][PID:2181572] Releasing VM back to pool [2025-07-10 12:54:04,052][ INFO][PID:2181572] Searching for 'success' file in resultdir [2025-07-10 12:54:04,053][ INFO][PID:2181572] Going to sign pkgs from source: 9262832-fedora-41-ppc64le in chroot: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le [2025-07-10 12:54:04,053][ INFO][PID:2181572] Calling '/bin/sign -u nmstate#nmstate-git@copr.fedorahosted.org -p' (attempt #1) [2025-07-10 12:54:04,379][ INFO][PID:2181572] Calling '/bin/sign -4 -h sha256 -u nmstate#nmstate-git@copr.fedorahosted.org -r /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-static-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm' (attempt #1) [2025-07-10 12:54:05,475][ INFO][PID:2181572] signed rpm: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-static-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm [2025-07-10 12:54:05,476][ INFO][PID:2181572] Calling '/bin/sign -4 -h sha256 -u nmstate#nmstate-git@copr.fedorahosted.org -r /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-libs-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm' (attempt #1) [2025-07-10 12:54:06,414][ INFO][PID:2181572] signed rpm: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-libs-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm [2025-07-10 12:54:06,415][ INFO][PID:2181572] Calling '/bin/sign -4 -h sha256 -u nmstate#nmstate-git@copr.fedorahosted.org -r /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-devel-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm' (attempt #1) [2025-07-10 12:54:07,286][ INFO][PID:2181572] signed rpm: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-devel-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm [2025-07-10 12:54:07,287][ INFO][PID:2181572] Calling '/bin/sign -4 -h sha256 -u nmstate#nmstate-git@copr.fedorahosted.org -r /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/python3-libnmstate-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm' (attempt #1) [2025-07-10 12:54:08,159][ INFO][PID:2181572] signed rpm: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/python3-libnmstate-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm [2025-07-10 12:54:08,160][ INFO][PID:2181572] Calling '/bin/sign -4 -h sha256 -u nmstate#nmstate-git@copr.fedorahosted.org -r /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-2.2.48-0.20250710.2581git13e44aa8.fc41.src.rpm' (attempt #1) [2025-07-10 12:54:09,454][ INFO][PID:2181572] signed rpm: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-2.2.48-0.20250710.2581git13e44aa8.fc41.src.rpm [2025-07-10 12:54:09,455][ INFO][PID:2181572] Calling '/bin/sign -4 -h sha256 -u nmstate#nmstate-git@copr.fedorahosted.org -r /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm' (attempt #1) [2025-07-10 12:54:10,402][ INFO][PID:2181572] signed rpm: /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/nmstate-2.2.48-0.20250710.2581git13e44aa8.fc41.ppc64le.rpm [2025-07-10 12:54:10,403][ INFO][PID:2181572] Sign done [2025-07-10 12:54:10,403][ INFO][PID:2181572] Incremental createrepo run, adding 09262832-nmstate into https://download.copr.fedorainfracloud.org/results/nmstate/nmstate-git/fedora-41-ppc64le, (auto-create-repo=True) [2025-07-10 12:54:10,405][ INFO][PID:2181572] Running command 'copr-repo --batched /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le --add 09262832-nmstate --no-appstream-metadata' as PID 2297063 [2025-07-10 12:54:11,143][ INFO][PID:2181572] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le --add 09262832-nmstate --no-appstream-metadata) [2025-07-10 12:54:11,143][ INFO][PID:2181572] Getting build details [2025-07-10 12:54:11,144][ INFO][PID:2181572] Listing built binary packages in /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate [2025-07-10 12:54:11,144][ INFO][PID:2181572] Built packages: ['nmstate-libs 2.2.48', 'nmstate-devel 2.2.48', 'python3-libnmstate 2.2.48', 'nmstate 2.2.48', 'nmstate-static 2.2.48'] [2025-07-10 12:54:11,145][ INFO][PID:2181572] build details: {'built_packages': 'nmstate-libs 2.2.48\nnmstate-devel 2.2.48\npython3-libnmstate 2.2.48\nnmstate 2.2.48\nnmstate-static 2.2.48'} [2025-07-10 12:54:11,145][ INFO][PID:2181572] Retrieving pubkey [2025-07-10 12:54:11,145][ INFO][PID:2181572] Calling '/bin/sign -u nmstate#nmstate-git@copr.fedorahosted.org -p' (attempt #1) [2025-07-10 12:54:11,481][ INFO][PID:2181572] Added pubkey for user nmstate project nmstate-git into: /var/lib/copr/public_html/results/nmstate/nmstate-git/pubkey.gpg [2025-07-10 12:54:11,481][ INFO][PID:2181572] Finished build: id=9262832 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/nmstate/nmstate-git chroot=fedora-41-ppc64le [2025-07-10 12:54:11,483][ INFO][PID:2181572] Worker succeeded build, took 617.8362293243408 [2025-07-10 12:54:11,484][ INFO][PID:2181572] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "nmstate", "project_name": "nmstate-git", "project_dirname": "nmstate-git", "submitter": "https://api.github.com/users/cathay4t", "ended_on": 1752152051.4834807, "started_on": 1752151433.6472514, "submitted_on": null, "status": 1, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9262832-fedora-41-ppc64le", "build_id": 9262832, "package_name": "nmstate", "package_version": "2.2.48-0.20250710.2581git13e44aa8", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/nmstate/nmstate-git/nmstate", "git_hash": "80378efcd5de8840356f3d64b82f378ad7f29b11", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "nmstate/nmstate-git--https://api.github.com/users/cathay4t", "results": { "packages": [ { "name": "nmstate", "epoch": null, "version": "2.2.48", "release": "0.20250710.2581git13e44aa8.fc41", "arch": "src" }, { "name": "nmstate-libs", "epoch": null, "version": "2.2.48", "release": "0.20250710.2581git13e44aa8.fc41", "arch": "ppc64le" }, { "name": "nmstate-devel", "epoch": null, "version": "2.2.48", "release": "0.20250710.2581git13e44aa8.fc41", "arch": "ppc64le" }, { "name": "python3-libnmstate", "epoch": null, "version": "2.2.48", "release": "0.20250710.2581git13e44aa8.fc41", "arch": "ppc64le" }, { "name": "nmstate", "epoch": null, "version": "2.2.48", "release": "0.20250710.2581git13e44aa8.fc41", "arch": "ppc64le" }, { "name": "nmstate-static", "epoch": null, "version": "2.2.48", "release": "0.20250710.2581git13e44aa8.fc41", "arch": "ppc64le" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/nmstate/nmstate-git/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/nmstate/nmstate-git", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/nmstate/nmstate-git", "result_dir": "09262832-nmstate", "built_packages": "nmstate-libs 2.2.48\nnmstate-devel 2.2.48\npython3-libnmstate 2.2.48\nnmstate 2.2.48\nnmstate-static 2.2.48", "id": 9262832, "mockchain_macros": { "copr_username": "nmstate", "copr_projectname": "nmstate-git", "vendor": "Fedora Project COPR (nmstate/nmstate-git)" } } ] } [2025-07-10 12:54:11,546][ INFO][PID:2181572] Sending fedora-messaging bus message in build.end [2025-07-10 12:54:11,570][ INFO][PID:2181572] Compressing /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/builder-live.log by gzip [2025-07-10 12:54:11,571][ INFO][PID:2181572] Running command 'gzip /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/builder-live.log' as PID 2297145 [2025-07-10 12:54:11,580][ INFO][PID:2181572] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/builder-live.log) [2025-07-10 12:54:11,581][ INFO][PID:2181572] Compressing /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/backend.log by gzip [2025-07-10 12:54:11,582][ INFO][PID:2181572] Running command 'gzip /var/lib/copr/public_html/results/nmstate/nmstate-git/fedora-41-ppc64le/09262832-nmstate/backend.log' as PID 2297146