[2025-12-12 09:23:16,235][  INFO][PID:2891934] Marking build as starting
[2025-12-12 09:23:16,517][  INFO][PID:2891934] Checking that /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/repodata/repomd.xml exists
[2025-12-12 09:23:16,529][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:23:16,530][  INFO][PID:2891934] VM allocation process starts
[2025-12-12 09:23:16,539][  INFO][PID:2891934] Trying to allocate VM: ResallocHost, ticket_id=300308, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-12 09:23:19,552][  INFO][PID:2891934] Allocated host ResallocHost, ticket_id=300308, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00491515_20251212_091312, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-12 09:23:19,552][  INFO][PID:2891934] Allocating ssh connection to builder
[2025-12-12 09:23:19,553][  INFO][PID:2891934] Checking that builder machine is OK
[2025-12-12 09:24:19,621][ ERROR][PID:2891934] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18c port 22: Connection timed out

[2025-12-12 09:24:29,622][ ERROR][PID:2891934] Re-try request for task on 'ResallocHost, ticket_id=300308, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00491515_20251212_091312, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-12 09:24:29,622][  INFO][PID:2891934] Releasing VM back to pool
[2025-12-12 09:24:29,646][  INFO][PID:2891934] Retry #1 (on other host)
[2025-12-12 09:24:29,647][  INFO][PID:2891934] Checking that /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/repodata/repomd.xml exists
[2025-12-12 09:24:29,648][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:24:29,648][  INFO][PID:2891934] VM allocation process starts
[2025-12-12 09:24:29,671][  INFO][PID:2891934] Trying to allocate VM: ResallocHost, ticket_id=300327, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-12 09:24:34,672][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:24:35,705][  INFO][PID:2891934] Allocated host ResallocHost, ticket_id=300327, hostname=2620:52:6:1161:dead:beef:cafe:c108, name=vmhost_p09_02_prod_00491585_20251212_091548, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-12 09:24:35,706][  INFO][PID:2891934] Allocating ssh connection to builder
[2025-12-12 09:24:35,706][  INFO][PID:2891934] Checking that builder machine is OK
[2025-12-12 09:25:35,762][ ERROR][PID:2891934] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c108 port 22: Connection timed out

[2025-12-12 09:25:45,763][ ERROR][PID:2891934] Re-try request for task on 'ResallocHost, ticket_id=300327, hostname=2620:52:6:1161:dead:beef:cafe:c108, name=vmhost_p09_02_prod_00491585_20251212_091548, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-12 09:25:45,763][  INFO][PID:2891934] Releasing VM back to pool
[2025-12-12 09:25:45,774][  INFO][PID:2891934] Retry #2 (on other host)
[2025-12-12 09:25:45,775][  INFO][PID:2891934] Checking that /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/repodata/repomd.xml exists
[2025-12-12 09:25:45,775][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:25:45,776][  INFO][PID:2891934] VM allocation process starts
[2025-12-12 09:25:45,790][  INFO][PID:2891934] Trying to allocate VM: ResallocHost, ticket_id=300333, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-12 09:25:50,791][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:25:51,810][  INFO][PID:2891934] Allocated host ResallocHost, ticket_id=300333, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_00491644_20251212_091903, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-12 09:25:51,811][  INFO][PID:2891934] Allocating ssh connection to builder
[2025-12-12 09:25:51,812][  INFO][PID:2891934] Checking that builder machine is OK
[2025-12-12 09:25:52,785][  INFO][PID:2891934] Running remote command: copr-builder-ready rhel+epel-10-ppc64le
[2025-12-12 09:25:53,307][  INFO][PID:2891934] Red Hat subscription not needed for rhel+epel-10-ppc64le
Builder is ready to be used

[2025-12-12 09:25:53,309][  INFO][PID:2891934] Filling build.info file with builder info
[2025-12-12 09:25:53,311][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:25:53,312][  INFO][PID:2891934] Not re-notifying FE and msg buses for the new host.
[2025-12-12 09:25:53,313][  INFO][PID:2891934] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9901906-rhel+epel-10-ppc64le --chroot rhel+epel-10-ppc64le --detached
[2025-12-12 09:25:54,251][  INFO][PID:2891934] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c195' (ED25519) to the list of known hosts.


[2025-12-12 09:25:54,252][  INFO][PID:2891934] Downloading the builder-live.log file, attempt 1
[2025-12-12 09:25:54,253][  INFO][PID:2891934] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c195 copr-rpmbuild-log
[2025-12-12 09:25:59,263][  INFO][PID:2891934] Periodic builder liveness probe: alive
[2025-12-12 09:25:59,263][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:04,264][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:09,265][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:14,267][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:19,267][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:24,268][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:29,269][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:34,270][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:39,272][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:44,273][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:49,274][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:54,275][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:26:59,276][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:04,277][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:09,278][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:14,279][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:19,280][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:24,281][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:29,282][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:34,283][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:39,284][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:44,285][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:49,286][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:54,287][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:27:59,288][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:04,289][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:09,290][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:14,290][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:19,292][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:19,368][ ERROR][PID:2891934] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-12 09:28:24,292][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:29,293][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:29,369][WARNING][PID:2891934] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-12 09:28:34,294][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:39,295][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:39,370][  INFO][PID:2891934] Downloading the builder-live.log file, attempt 2
[2025-12-12 09:28:39,371][  INFO][PID:2891934] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c195 copr-rpmbuild-log
[2025-12-12 09:28:44,296][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:49,297][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:54,298][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:28:59,299][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:04,300][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:09,300][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:14,301][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:19,302][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:24,303][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:29,304][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:34,305][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:39,306][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:44,307][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:49,308][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:54,309][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:29:59,310][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:04,311][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:09,312][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:14,313][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:19,314][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:24,315][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:29,316][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:34,317][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:39,318][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:44,319][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:49,320][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:54,321][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:30:59,322][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:04,323][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:09,324][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:14,325][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:19,326][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:24,327][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:29,328][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:34,329][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:39,330][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:44,331][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:49,332][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:54,333][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:31:59,334][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:04,335][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:09,336][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:14,337][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:19,339][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:24,340][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:29,341][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:34,343][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:39,344][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:44,345][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:49,346][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:54,347][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:32:59,348][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:04,349][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:09,350][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:14,351][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:19,352][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:24,353][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:29,354][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:34,356][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:39,356][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:44,358][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:49,359][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:54,360][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:33:59,361][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:04,362][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:09,363][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:14,364][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:19,365][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:24,365][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:29,366][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:34,367][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:39,368][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:44,369][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:49,370][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:54,372][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:34:59,372][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:04,373][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:09,374][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:14,375][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:19,376][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:24,378][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:29,379][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:34,380][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:39,381][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:44,382][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:49,383][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:54,384][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:35:59,392][  INFO][PID:2891934] Periodic builder liveness probe: alive
[2025-12-12 09:35:59,393][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:04,393][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:09,394][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:14,395][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:19,397][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:24,398][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:29,400][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:34,401][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:39,403][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:44,404][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:49,406][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:54,408][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:36:59,410][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:04,411][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:09,412][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:14,413][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:19,414][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:24,415][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:29,416][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:34,417][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:39,419][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:44,420][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:49,421][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:54,422][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:37:59,423][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:04,424][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:09,425][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:14,426][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:19,427][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:24,428][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:29,429][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:34,430][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:39,431][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:44,432][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:49,433][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:54,439][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:38:59,443][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:04,444][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:09,445][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:14,446][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:19,447][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:24,447][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:29,448][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:34,449][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:39,450][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:44,451][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:49,452][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:54,453][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:39:59,454][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:04,455][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:09,456][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:14,457][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:19,458][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:24,460][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:29,461][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:34,462][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:39,463][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:44,464][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:49,465][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:54,466][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:40:59,467][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:04,468][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:09,468][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:14,469][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:19,470][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:24,471][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:29,472][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:34,473][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:39,474][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:44,475][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:49,476][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:54,477][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:41:59,478][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:04,479][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:09,480][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:14,481][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:19,482][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:24,483][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:29,484][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:34,485][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:39,486][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:44,487][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:49,488][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:54,489][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:42:59,490][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:04,491][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:09,492][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:14,493][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:19,494][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:24,495][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:29,496][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:34,496][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:39,498][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:44,499][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:49,499][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:54,504][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:43:59,505][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:04,507][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:09,508][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:14,509][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:19,510][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:24,511][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:29,512][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:34,513][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:39,514][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:44,515][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:49,516][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:54,518][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:44:59,519][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:04,520][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:09,521][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:14,522][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:19,523][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:24,524][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:29,525][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:34,526][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:39,527][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:44,528][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:49,529][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:54,530][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:45:59,538][  INFO][PID:2891934] Periodic builder liveness probe: alive
[2025-12-12 09:45:59,538][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:04,539][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:09,540][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:14,542][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:19,543][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:24,544][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:29,545][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:34,546][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:39,547][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:44,549][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:49,550][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:54,551][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:46:59,552][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:04,553][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:09,554][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:14,555][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:19,556][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:24,557][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:29,558][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:34,559][  INFO][PID:2891934] Checking for cancel request
[2025-12-12 09:47:36,714][  INFO][PID:2891934] Downloading results from builder
[2025-12-12 09:47:36,714][  INFO][PID:2891934] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c195]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf started
[2025-12-12 09:47:36,715][  INFO][PID:2891934] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c195]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/ &> /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/build-09901906.rsync.log
[2025-12-12 09:48:36,784][  INFO][PID:2891934] rsyncing finished.
[2025-12-12 09:48:36,784][ ERROR][PID:2891934] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-12 09:48:36,785][ ERROR][PID:2891934] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-12 09:48:46,785][  INFO][PID:2891934] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c195]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf started
[2025-12-12 09:48:46,787][  INFO][PID:2891934] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c195]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/ &> /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/build-09901906.rsync.log
[2025-12-12 09:49:46,863][  INFO][PID:2891934] rsyncing finished.
[2025-12-12 09:49:46,864][ ERROR][PID:2891934] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-12 09:49:46,865][ ERROR][PID:2891934] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-12 09:49:56,865][  INFO][PID:2891934] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c195]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf started
[2025-12-12 09:49:56,866][  INFO][PID:2891934] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c195]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/ &> /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/build-09901906.rsync.log
[2025-12-12 09:50:56,942][  INFO][PID:2891934] rsyncing finished.
[2025-12-12 09:50:56,943][ ERROR][PID:2891934] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-12 09:50:56,943][ ERROR][PID:2891934] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-12 09:51:06,944][ ERROR][PID:2891934] Re-try request for task on 'ResallocHost, ticket_id=300333, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_00491644_20251212_091903, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts
[2025-12-12 09:51:06,944][  INFO][PID:2891934] Releasing VM back to pool
[2025-12-12 09:51:06,959][ ERROR][PID:2891934] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c18c', '2620:52:6:1161:dead:beef:cafe:c108', '2620:52:6:1161:dead:beef:cafe:c195'}
[2025-12-12 09:51:06,960][WARNING][PID:2891934] Switching not-finished job state to 'failed'
[2025-12-12 09:51:06,961][  INFO][PID:2891934] Worker failed build, took 1513.6479861736298
[2025-12-12 09:51:06,961][  INFO][PID:2891934] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "mjg",
            "project_name": "mupdf-git",
            "project_dirname": "mupdf-git",
            "submitter": "https://api.github.com/users/mjg",
            "ended_on": 1765533066.9605012,
            "started_on": 1765531553.312515,
            "submitted_on": null,
            "status": 0,
            "chroot": "rhel+epel-10-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9901906-rhel+epel-10-ppc64le",
            "build_id": 9901906,
            "package_name": "mupdf",
            "package_version": "1.27.0~rc1^17.gd3b755657-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/mjg/mupdf-git/mupdf",
            "git_hash": "ad4c2a821d8de54421596e325ff03a2eec71ef65",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "mjg/mupdf-git--https://api.github.com/users/mjg",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/mjg/mupdf-git/rhel+epel-10-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/mjg/mupdf-git",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/mjg/mupdf-git",
            "result_dir": "09901906-mupdf",
            "built_packages": "",
            "id": 9901906,
            "mockchain_macros": {
                "copr_username": "mjg",
                "copr_projectname": "mupdf-git",
                "vendor": "Fedora Project COPR (mjg/mupdf-git)"
            }
        }
    ]
}
[2025-12-12 09:51:07,033][  INFO][PID:2891934] Sending fedora-messaging bus message in build.end
[2025-12-12 09:51:07,598][  INFO][PID:2891934] Compressing /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/builder-live.log by gzip
[2025-12-12 09:51:07,600][  INFO][PID:2891934] Running command 'gzip /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/builder-live.log' as PID 3032601
[2025-12-12 09:51:07,661][  INFO][PID:2891934] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/builder-live.log)
[2025-12-12 09:51:07,662][  INFO][PID:2891934] Compressing /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/backend.log by gzip
[2025-12-12 09:51:07,663][  INFO][PID:2891934] Running command 'gzip /var/lib/copr/public_html/results/mjg/mupdf-git/rhel+epel-10-ppc64le/09901906-mupdf/backend.log' as PID 3032602