[2026-01-02 00:14:34,596][ INFO][PID:1840296] Marking build as starting [2026-01-02 00:14:34,680][ INFO][PID:1840296] Checking that /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/repodata/repomd.xml exists [2026-01-02 00:14:34,681][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:14:34,682][ INFO][PID:1840296] VM allocation process starts [2026-01-02 00:14:34,696][ INFO][PID:1840296] Trying to allocate VM: ResallocHost, ticket_id=619626, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-02 00:14:37,709][ INFO][PID:1840296] Allocated host ResallocHost, ticket_id=619626, hostname=2620:52:6:1161:dead:beef:cafe:c18e, name=vmhost_p09_04_prod_00885333_20260101_231739, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-02 00:14:37,709][ INFO][PID:1840296] Allocating ssh connection to builder [2026-01-02 00:14:37,710][ INFO][PID:1840296] Checking that builder machine is OK [2026-01-02 00:15:37,777][ ERROR][PID:1840296] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18e port 22: Connection timed out [2026-01-02 00:15:47,778][ ERROR][PID:1840296] Re-try request for task on 'ResallocHost, ticket_id=619626, hostname=2620:52:6:1161:dead:beef:cafe:c18e, name=vmhost_p09_04_prod_00885333_20260101_231739, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2026-01-02 00:15:47,778][ INFO][PID:1840296] Releasing VM back to pool [2026-01-02 00:15:47,788][ INFO][PID:1840296] Retry #1 (on other host) [2026-01-02 00:15:47,788][ INFO][PID:1840296] Checking that /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/repodata/repomd.xml exists [2026-01-02 00:15:47,789][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:15:47,789][ INFO][PID:1840296] VM allocation process starts [2026-01-02 00:15:47,800][ INFO][PID:1840296] Trying to allocate VM: ResallocHost, ticket_id=619633, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-02 00:15:50,811][ INFO][PID:1840296] Allocated host ResallocHost, ticket_id=619633, hostname=2620:52:6:1161:dead:beef:cafe:c192, name=vmhost_p09_04_prod_00885352_20260101_232224, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-02 00:15:50,811][ INFO][PID:1840296] Allocating ssh connection to builder [2026-01-02 00:15:50,812][ INFO][PID:1840296] Checking that builder machine is OK [2026-01-02 00:15:51,773][ INFO][PID:1840296] Running remote command: copr-builder-ready epel-10-ppc64le [2026-01-02 00:15:52,284][ INFO][PID:1840296] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-02 00:15:52,285][ INFO][PID:1840296] Filling build.info file with builder info [2026-01-02 00:15:52,285][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:15:52,286][ INFO][PID:1840296] Not re-notifying FE and msg buses for the new host. [2026-01-02 00:15:52,286][ INFO][PID:1840296] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9968654-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2026-01-02 00:15:53,265][ INFO][PID:1840296] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c192' (ED25519) to the list of known hosts. [2026-01-02 00:15:53,266][ INFO][PID:1840296] Downloading the builder-live.log file, attempt 1 [2026-01-02 00:15:53,267][ INFO][PID:1840296] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c192 copr-rpmbuild-log [2026-01-02 00:15:58,271][ INFO][PID:1840296] Periodic builder liveness probe: alive [2026-01-02 00:15:58,271][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:03,272][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:08,273][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:13,274][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:18,275][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:23,276][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:28,277][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:33,278][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:38,278][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:43,280][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:48,281][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:53,281][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:16:58,282][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:03,283][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:08,284][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:13,285][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:18,286][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:23,287][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:28,288][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:33,289][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:38,290][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:43,291][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:48,292][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:53,293][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:17:58,294][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:03,295][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:08,296][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:13,297][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:18,297][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:23,298][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:28,299][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:33,300][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:38,301][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:43,302][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:48,303][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:53,304][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:18:58,305][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:03,305][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:08,306][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:13,307][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:18,308][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:23,309][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:28,310][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:33,311][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:38,312][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:43,312][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:48,313][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:53,314][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:19:58,315][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:03,316][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:08,317][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:13,318][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:18,319][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:23,320][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:28,321][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:33,321][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:38,322][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:43,323][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:48,324][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:53,325][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:20:58,326][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:03,327][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:08,328][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:13,329][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:18,330][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:23,331][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:28,331][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:33,332][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:38,333][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:43,334][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:48,335][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:53,336][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:21:58,337][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:03,338][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:08,339][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:13,339][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:18,340][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:23,341][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:28,342][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:33,343][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:38,344][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:43,345][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:48,346][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:53,347][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:22:58,348][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:03,348][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:08,349][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:13,350][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:18,351][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:23,352][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:28,353][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:33,354][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:38,355][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:43,356][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:48,357][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:53,357][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:23:58,358][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:03,359][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:08,360][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:13,405][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:18,406][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:23,450][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:28,451][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:33,452][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:38,721][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:43,722][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:48,723][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:53,724][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:24:58,725][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:03,726][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:08,727][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:13,728][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:18,817][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:23,818][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:28,819][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:33,820][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:38,821][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:43,822][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:49,039][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:54,041][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:25:59,048][ INFO][PID:1840296] Periodic builder liveness probe: alive [2026-01-02 00:25:59,048][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:04,049][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:09,050][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:14,051][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:19,052][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:24,053][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:29,054][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:34,055][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:39,055][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:44,056][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:49,057][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:54,058][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:26:59,059][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:04,060][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:09,061][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:14,062][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:19,063][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:24,064][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:29,065][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:34,065][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:39,066][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:44,067][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:49,068][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:54,181][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:27:59,182][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:04,184][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:09,186][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:14,187][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:19,188][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:24,191][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:29,192][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:34,193][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:39,193][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:44,194][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:49,195][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:54,196][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:28:59,197][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:04,198][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:09,199][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:14,199][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:19,200][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:24,237][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:29,238][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:34,239][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:39,240][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:44,241][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:49,242][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:54,244][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:29:59,245][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:04,387][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:09,389][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:14,389][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:19,390][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:24,391][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:29,393][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:34,394][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:39,395][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:44,397][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:49,398][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:54,399][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:30:59,400][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:04,401][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:09,402][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:14,403][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:19,404][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:24,405][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:29,406][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:34,407][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:39,408][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:44,409][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:49,410][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:54,412][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:31:59,413][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:04,414][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:09,415][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:14,416][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:19,417][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:24,418][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:29,419][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:34,449][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:39,451][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:44,452][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:49,614][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:54,615][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:32:59,616][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:04,617][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:09,619][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:14,619][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:19,620][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:24,621][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:29,936][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:34,938][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:39,939][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:44,940][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:49,941][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:54,942][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:33:59,943][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:04,944][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:09,945][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:14,946][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:19,947][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:24,947][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:29,948][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:34,950][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:39,951][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:44,952][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:49,953][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:54,954][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:34:59,955][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:04,955][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:09,956][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:14,957][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:19,958][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:24,959][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:29,960][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:34,961][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:39,962][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:44,963][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:49,964][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:54,965][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:35:59,971][ INFO][PID:1840296] Periodic builder liveness probe: alive [2026-01-02 00:35:59,971][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:04,972][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:09,973][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:14,973][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:19,974][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:25,032][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:30,033][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:35,034][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:40,035][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:45,037][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:50,038][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:36:55,039][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:00,041][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:05,042][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:10,043][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:15,044][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:20,045][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:25,046][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:30,090][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:35,371][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:40,372][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:45,373][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:50,374][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:37:55,375][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:00,376][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:05,377][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:10,378][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:15,379][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:20,380][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:25,381][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:30,382][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:35,383][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:40,384][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:45,385][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:50,386][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:38:55,386][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:00,387][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:05,428][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:10,429][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:15,430][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:20,576][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:25,937][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:30,938][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:35,939][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:40,940][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:45,941][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:50,942][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:39:55,942][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:00,943][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:05,944][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:10,945][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:15,946][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:20,947][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:25,948][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:30,949][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:35,949][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:40,950][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:45,952][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:50,957][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:40:55,958][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:00,959][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:05,960][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:10,961][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:15,962][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:20,963][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:25,964][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:30,965][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:35,966][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:40,967][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:45,981][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:50,982][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:41:55,983][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:00,984][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:05,985][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:10,986][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:15,987][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:20,988][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:25,988][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:30,989][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:35,990][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:40,992][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:46,095][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:51,097][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:42:56,098][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:01,099][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:06,100][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:11,176][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:16,178][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:21,223][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:26,224][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:31,225][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:36,226][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:41,227][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:46,228][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:51,229][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:43:56,230][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:01,230][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:06,231][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:11,232][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:16,233][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:21,234][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:26,235][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:31,236][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:36,237][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:41,238][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:46,239][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:51,240][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:44:56,347][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:01,348][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:06,351][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:11,352][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:16,353][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:21,354][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:26,355][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:31,355][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:36,356][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:41,357][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:46,358][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:51,359][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:45:56,359][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:46:01,368][ INFO][PID:1840296] Periodic builder liveness probe: dead [2026-01-02 00:46:01,369][ INFO][PID:1840296] calling cancel callback, and waiting [2026-01-02 00:46:08,480][ INFO][PID:1840296] Cancel request succeeded out: success err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c192' (ED25519) to the list of known hosts. [2026-01-02 00:46:08,481][ ERROR][PID:1840296] Re-try request for task on 'ResallocHost, ticket_id=619633, hostname=2620:52:6:1161:dead:beef:cafe:c192, name=vmhost_p09_04_prod_00885352_20260101_232224, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Resalloc ticket FAILED [2026-01-02 00:46:08,481][ INFO][PID:1840296] Releasing VM back to pool [2026-01-02 00:46:08,491][ INFO][PID:1840296] Retry #2 (on other host) [2026-01-02 00:46:08,492][ INFO][PID:1840296] Checking that /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/repodata/repomd.xml exists [2026-01-02 00:46:08,493][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:46:08,493][ INFO][PID:1840296] VM allocation process starts [2026-01-02 00:46:08,540][ INFO][PID:1840296] Trying to allocate VM: ResallocHost, ticket_id=619810, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-02 00:46:11,595][ INFO][PID:1840296] Allocated host ResallocHost, ticket_id=619810, hostname=2620:52:6:1161:dead:beef:cafe:c14e, name=vmhost_p09_03_prod_00885411_20260101_234923, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-02 00:46:11,595][ INFO][PID:1840296] Allocating ssh connection to builder [2026-01-02 00:46:11,596][ INFO][PID:1840296] Checking that builder machine is OK [2026-01-02 00:46:12,499][ INFO][PID:1840296] Running remote command: copr-builder-ready epel-10-ppc64le [2026-01-02 00:46:12,993][ INFO][PID:1840296] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-02 00:46:12,994][ INFO][PID:1840296] Filling build.info file with builder info [2026-01-02 00:46:13,047][ INFO][PID:1840296] Checking for cancel request [2026-01-02 00:46:13,049][ INFO][PID:1840296] Not re-notifying FE and msg buses for the new host. [2026-01-02 00:46:13,050][ INFO][PID:1840296] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9968654-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2026-01-02 00:46:13,861][ INFO][PID:1840296] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14e' (ED25519) to the list of known hosts. [2026-01-02 00:46:13,862][ INFO][PID:1840296] Downloading the builder-live.log file, attempt 1 [2026-01-02 00:46:13,888][ INFO][PID:1840296] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14e copr-rpmbuild-log [2026-01-02 00:46:18,862][ INFO][PID:1840296] calling cancel callback, and waiting [2026-01-02 00:47:18,931][ ERROR][PID:1840296] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14e port 22: Connection timed out [2026-01-02 00:48:29,000][ ERROR][PID:1840296] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14e port 22: Connection timed out [2026-01-02 00:48:42,670][ INFO][PID:1840296] Cancel request succeeded out: success err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14e' (ED25519) to the list of known hosts. [2026-01-02 00:48:42,670][ INFO][PID:1840296] Downloading results from builder [2026-01-02 00:48:42,671][ INFO][PID:1840296] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14e]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128 started [2026-01-02 00:48:42,671][ INFO][PID:1840296] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14e]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/ &> /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/build-09968654.rsync.log [2026-01-02 00:49:42,745][ INFO][PID:1840296] rsyncing finished. [2026-01-02 00:49:42,777][ ERROR][PID:1840296] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-02 00:49:42,778][ ERROR][PID:1840296] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-02 00:49:52,778][ INFO][PID:1840296] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14e]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128 started [2026-01-02 00:49:52,779][ INFO][PID:1840296] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14e]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/ &> /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/build-09968654.rsync.log [2026-01-02 00:50:52,862][ INFO][PID:1840296] rsyncing finished. [2026-01-02 00:50:52,862][ ERROR][PID:1840296] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-02 00:50:52,863][ ERROR][PID:1840296] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-02 00:51:02,863][ INFO][PID:1840296] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14e]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128 started [2026-01-02 00:51:02,873][ INFO][PID:1840296] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14e]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/ &> /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/build-09968654.rsync.log [2026-01-02 00:52:03,107][ INFO][PID:1840296] rsyncing finished. [2026-01-02 00:52:03,107][ ERROR][PID:1840296] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-02 00:52:03,108][ ERROR][PID:1840296] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-02 00:52:13,108][ ERROR][PID:1840296] Re-try request for task on 'ResallocHost, ticket_id=619810, hostname=2620:52:6:1161:dead:beef:cafe:c14e, name=vmhost_p09_03_prod_00885411_20260101_234923, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts [2026-01-02 00:52:13,109][ INFO][PID:1840296] Releasing VM back to pool [2026-01-02 00:52:13,117][ ERROR][PID:1840296] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c14e', '2620:52:6:1161:dead:beef:cafe:c18e', '2620:52:6:1161:dead:beef:cafe:c192'} [2026-01-02 00:52:13,118][WARNING][PID:1840296] Switching not-finished job state to 'failed' [2026-01-02 00:52:13,118][ INFO][PID:1840296] Worker failed build, took 360.0687425136566 [2026-01-02 00:52:13,119][ INFO][PID:1840296] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "yselkowitz", "project_name": "gnome-epel", "project_dirname": "gnome-epel", "submitter": "yselkowitz", "ended_on": 1767315133.1180222, "started_on": 1767314773.0492797, "submitted_on": null, "status": 0, "chroot": "epel-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9968654-epel-10-ppc64le", "build_id": 9968654, "package_name": "mozjs128", "package_version": "128.11.0-8", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/yselkowitz/gnome-epel/mozjs128", "git_hash": "e3a702e4c0aed408b20fd3dad99e2bb0fecbb380", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "yselkowitz/gnome-epel--yselkowitz", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/yselkowitz/gnome-epel/epel-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/yselkowitz/gnome-epel", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/yselkowitz/gnome-epel", "result_dir": "09968654-mozjs128", "built_packages": "", "id": 9968654, "mockchain_macros": { "copr_username": "yselkowitz", "copr_projectname": "gnome-epel", "vendor": "Fedora Project COPR (yselkowitz/gnome-epel)" } } ] } [2026-01-02 00:52:13,185][ INFO][PID:1840296] Sending fedora-messaging bus message in build.end [2026-01-02 00:52:13,725][ INFO][PID:1840296] Compressing /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/builder-live.log by gzip [2026-01-02 00:52:13,727][ INFO][PID:1840296] Running command 'gzip /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/builder-live.log' as PID 1956560 [2026-01-02 00:52:13,758][ INFO][PID:1840296] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/builder-live.log) [2026-01-02 00:52:13,759][ INFO][PID:1840296] Compressing /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/backend.log by gzip [2026-01-02 00:52:13,760][ INFO][PID:1840296] Running command 'gzip /var/lib/copr/public_html/results/yselkowitz/gnome-epel/epel-10-ppc64le/09968654-mozjs128/backend.log' as PID 1956563