[2025-12-14 01:42:46,338][  INFO][PID:2094220] Marking build as starting
[2025-12-14 01:42:46,398][  INFO][PID:2094220] Checking that /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/repodata/repomd.xml exists
[2025-12-14 01:42:46,399][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:42:46,400][  INFO][PID:2094220] VM allocation process starts
[2025-12-14 01:42:46,412][  INFO][PID:2094220] Trying to allocate VM: ResallocHost, ticket_id=326616, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-14 01:42:51,413][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:42:56,414][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:42:58,463][  INFO][PID:2094220] Allocated host ResallocHost, ticket_id=326616, hostname=2620:52:6:1161:dead:beef:cafe:c189, name=vmhost_p09_04_prod_00530876_20251214_013149, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-14 01:42:58,464][  INFO][PID:2094220] Allocating ssh connection to builder
[2025-12-14 01:42:58,464][  INFO][PID:2094220] Checking that builder machine is OK
[2025-12-14 01:43:58,533][ ERROR][PID:2094220] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c189 port 22: Connection timed out

[2025-12-14 01:44:08,534][ ERROR][PID:2094220] Re-try request for task on 'ResallocHost, ticket_id=326616, hostname=2620:52:6:1161:dead:beef:cafe:c189, name=vmhost_p09_04_prod_00530876_20251214_013149, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-14 01:44:08,534][  INFO][PID:2094220] Releasing VM back to pool
[2025-12-14 01:44:08,544][  INFO][PID:2094220] Retry #1 (on other host)
[2025-12-14 01:44:08,545][  INFO][PID:2094220] Checking that /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/repodata/repomd.xml exists
[2025-12-14 01:44:08,546][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:44:08,547][  INFO][PID:2094220] VM allocation process starts
[2025-12-14 01:44:08,557][  INFO][PID:2094220] Trying to allocate VM: ResallocHost, ticket_id=326630, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-14 01:44:13,558][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:44:14,585][  INFO][PID:2094220] Allocated host ResallocHost, ticket_id=326630, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00530877_20251214_013152, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-14 01:44:14,585][  INFO][PID:2094220] Allocating ssh connection to builder
[2025-12-14 01:44:14,586][  INFO][PID:2094220] Checking that builder machine is OK
[2025-12-14 01:44:15,482][  INFO][PID:2094220] Running remote command: copr-builder-ready epel-9-ppc64le
[2025-12-14 01:45:15,552][ ERROR][PID:2094220] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c191 port 22: Connection timed out

[2025-12-14 01:45:25,553][ ERROR][PID:2094220] Re-try request for task on 'ResallocHost, ticket_id=326630, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_00530877_20251214_013152, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-14 01:45:25,554][  INFO][PID:2094220] Releasing VM back to pool
[2025-12-14 01:45:25,567][  INFO][PID:2094220] Retry #2 (on other host)
[2025-12-14 01:45:25,568][  INFO][PID:2094220] Checking that /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/repodata/repomd.xml exists
[2025-12-14 01:45:25,568][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:25,568][  INFO][PID:2094220] VM allocation process starts
[2025-12-14 01:45:25,581][  INFO][PID:2094220] Trying to allocate VM: ResallocHost, ticket_id=326634, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-14 01:45:30,582][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:31,602][  INFO][PID:2094220] Allocated host ResallocHost, ticket_id=326634, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_00530921_20251214_013415, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-14 01:45:31,603][  INFO][PID:2094220] Allocating ssh connection to builder
[2025-12-14 01:45:31,603][  INFO][PID:2094220] Checking that builder machine is OK
[2025-12-14 01:45:32,503][  INFO][PID:2094220] Running remote command: copr-builder-ready epel-9-ppc64le
[2025-12-14 01:45:33,020][  INFO][PID:2094220] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-12-14 01:45:33,020][  INFO][PID:2094220] Filling build.info file with builder info
[2025-12-14 01:45:33,021][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:33,021][  INFO][PID:2094220] Not re-notifying FE and msg buses for the new host.
[2025-12-14 01:45:33,022][  INFO][PID:2094220] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9907080-epel-9-ppc64le --chroot epel-9-ppc64le --detached
[2025-12-14 01:45:33,907][  INFO][PID:2094220] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c116' (ED25519) to the list of known hosts.


[2025-12-14 01:45:33,908][  INFO][PID:2094220] Downloading the builder-live.log file, attempt 1
[2025-12-14 01:45:33,909][  INFO][PID:2094220] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c116 copr-rpmbuild-log
[2025-12-14 01:45:38,926][  INFO][PID:2094220] Periodic builder liveness probe: alive
[2025-12-14 01:45:38,927][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:43,928][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:48,929][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:53,930][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:45:58,931][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:03,932][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:08,933][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:13,934][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:18,935][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:23,936][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:28,937][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:33,938][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:38,939][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:43,940][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:48,941][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:53,942][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:46:58,943][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:03,944][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:08,944][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:13,945][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:18,946][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:23,947][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:28,948][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:33,949][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:38,950][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:43,952][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:48,953][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:53,954][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:47:58,955][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:03,958][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:08,960][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:13,961][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:18,962][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:23,977][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:29,280][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:34,281][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:39,282][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:44,283][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:49,285][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:54,286][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:48:59,287][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:04,289][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:09,290][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:14,291][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:19,292][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:24,293][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:29,294][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:34,295][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:39,296][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:44,297][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:49,298][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:54,299][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:49:59,300][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:04,301][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:09,302][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:14,303][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:19,304][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:24,306][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:29,307][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:34,308][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:39,309][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:44,312][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:49,314][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:54,314][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:50:59,316][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:04,317][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:09,318][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:14,319][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:19,320][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:24,321][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:29,537][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:34,537][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:39,539][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:44,540][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:49,541][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:54,603][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:51:59,604][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:04,637][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:09,714][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:14,715][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:19,716][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:24,729][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:29,730][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:34,731][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:39,732][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:44,733][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:49,734][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:54,735][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:52:59,736][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:04,737][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:09,738][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:14,739][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:19,740][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:24,741][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:29,742][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:34,743][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:39,744][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:44,745][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:49,746][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:54,747][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:53:59,769][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:04,770][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:09,771][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:14,772][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:19,773][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:24,774][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:29,844][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:34,845][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:39,846][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:44,847][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:49,848][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:54,849][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:54:59,850][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:04,851][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:09,852][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:14,853][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:19,854][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:24,855][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:29,856][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:34,857][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:39,866][  INFO][PID:2094220] Periodic builder liveness probe: alive
[2025-12-14 01:55:39,867][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:44,867][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:49,953][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:54,954][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:55:59,956][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:04,957][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:09,959][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:14,960][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:19,972][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:24,973][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:29,974][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:34,979][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:39,982][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:44,984][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:49,986][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:54,987][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:56:59,988][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:04,991][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:09,993][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:14,994][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:19,998][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:25,000][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:30,001][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:35,002][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:40,003][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:45,004][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:50,005][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:57:55,233][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:00,234][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:05,235][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:10,237][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:15,238][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:20,239][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:25,240][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:30,242][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:35,243][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:40,244][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:45,246][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:50,248][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:58:55,249][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:00,252][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:05,432][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:10,434][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:15,435][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:20,436][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:25,437][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:30,438][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:35,439][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:40,440][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:45,441][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:50,443][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 01:59:55,702][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:00,704][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:05,709][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:10,710][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:15,711][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:20,713][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:25,714][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:30,715][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:35,716][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:40,718][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:45,719][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:50,721][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:00:55,723][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:00,724][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:05,725][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:10,728][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:15,729][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:20,730][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:25,734][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:30,740][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:35,741][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:40,743][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:45,745][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:50,746][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:01:55,748][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:00,750][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:05,751][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:10,752][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:15,753][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:20,754][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:25,755][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:30,756][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:35,758][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:40,759][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:45,760][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:50,761][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:02:55,762][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:00,763][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:05,765][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:10,766][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:15,767][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:20,769][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:25,770][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:30,771][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:35,772][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:40,773][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:45,774][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:50,775][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:03:55,776][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:00,777][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:05,778][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:10,779][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:15,780][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:20,781][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:25,782][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:30,894][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:35,895][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:40,895][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:45,896][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:50,897][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:04:55,898][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:00,899][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:05,900][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:10,901][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:15,902][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:20,903][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:25,906][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:30,907][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:35,908][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:40,915][  INFO][PID:2094220] Periodic builder liveness probe: alive
[2025-12-14 02:05:40,915][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:45,916][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:50,917][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:05:55,918][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:00,919][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:05,920][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:10,921][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:15,922][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:20,924][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:25,925][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:30,925][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:35,926][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:40,927][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:45,928][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:50,929][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:06:55,930][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:00,931][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:05,932][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:10,934][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:15,935][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:20,936][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:25,937][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:30,938][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:35,939][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:40,940][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:45,942][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:50,943][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:07:55,949][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:00,951][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:05,957][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:10,958][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:15,959][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:20,960][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:25,962][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:30,965][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:35,988][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:40,991][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:45,994][  INFO][PID:2094220] Checking for cancel request
[2025-12-14 02:08:47,937][  INFO][PID:2094220] Downloading results from builder
[2025-12-14 02:08:47,968][  INFO][PID:2094220] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c116]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu started
[2025-12-14 02:08:48,006][  INFO][PID:2094220] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c116]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/ &> /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/build-09907080.rsync.log
[2025-12-14 02:09:48,031][  INFO][PID:2094220] rsyncing finished.
[2025-12-14 02:09:48,063][ ERROR][PID:2094220] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-14 02:09:48,071][ ERROR][PID:2094220] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-14 02:09:58,079][  INFO][PID:2094220] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c116]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu started
[2025-12-14 02:09:58,084][  INFO][PID:2094220] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c116]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/ &> /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/build-09907080.rsync.log
[2025-12-14 02:10:58,167][  INFO][PID:2094220] rsyncing finished.
[2025-12-14 02:10:58,167][ ERROR][PID:2094220] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-14 02:10:58,169][ ERROR][PID:2094220] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-14 02:11:08,175][  INFO][PID:2094220] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c116]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu started
[2025-12-14 02:11:08,177][  INFO][PID:2094220] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c116]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/ &> /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/build-09907080.rsync.log
[2025-12-14 02:12:08,213][  INFO][PID:2094220] rsyncing finished.
[2025-12-14 02:12:08,214][ ERROR][PID:2094220] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-14 02:12:08,214][ ERROR][PID:2094220] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-14 02:12:18,230][ ERROR][PID:2094220] Re-try request for task on 'ResallocHost, ticket_id=326634, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_00530921_20251214_013415, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts
[2025-12-14 02:12:18,238][  INFO][PID:2094220] Releasing VM back to pool
[2025-12-14 02:12:18,522][ ERROR][PID:2094220] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c189', '2620:52:6:1161:dead:beef:cafe:c191', '2620:52:6:1161:dead:beef:cafe:c116'}
[2025-12-14 02:12:18,542][WARNING][PID:2094220] Switching not-finished job state to 'failed'
[2025-12-14 02:12:18,545][  INFO][PID:2094220] Worker failed build, took 1605.5041892528534
[2025-12-14 02:12:18,579][  INFO][PID:2094220] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "ligenix",
            "project_name": "enterprise-cosmic-testing",
            "project_dirname": "enterprise-cosmic-testing",
            "submitter": "ligenix",
            "ended_on": 1765678338.526144,
            "started_on": 1765676733.0219548,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-9-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9907080-epel-9-ppc64le",
            "build_id": 9907080,
            "package_name": "cosmic-ext-applet-places-menu",
            "package_version": "0.1.0^git20250315.4430ee4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ligenix/enterprise-cosmic-testing/cosmic-ext-applet-places-menu",
            "git_hash": "feadca857d9092f9d5c3cdd37edbbd251eb8ea26",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "ligenix/enterprise-cosmic-testing--ligenix",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/ligenix/enterprise-cosmic/epel-9-ppc64le/",
                    "id": "copr_ligenix_enterprise_cosmic_epel_releasever_basearch",
                    "name": "Additional repo copr_ligenix_enterprise_cosmic_epel_releasever_basearch"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ligenix/enterprise-cosmic-testing",
            "result_dir": "09907080-cosmic-ext-applet-places-menu",
            "built_packages": "",
            "id": 9907080,
            "mockchain_macros": {
                "copr_username": "ligenix",
                "copr_projectname": "enterprise-cosmic-testing",
                "vendor": "Fedora Project COPR (ligenix/enterprise-cosmic-testing)"
            }
        }
    ]
}
[2025-12-14 02:12:19,716][  INFO][PID:2094220] Sending fedora-messaging bus message in build.end
[2025-12-14 02:12:22,845][  INFO][PID:2094220] Compressing /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/builder-live.log by gzip
[2025-12-14 02:12:22,847][  INFO][PID:2094220] Running command 'gzip /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/builder-live.log' as PID 2236163
[2025-12-14 02:12:23,109][  INFO][PID:2094220] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/builder-live.log)
[2025-12-14 02:12:23,109][  INFO][PID:2094220] Compressing /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/backend.log by gzip
[2025-12-14 02:12:23,110][  INFO][PID:2094220] Running command 'gzip /var/lib/copr/public_html/results/ligenix/enterprise-cosmic-testing/epel-9-ppc64le/09907080-cosmic-ext-applet-places-menu/backend.log' as PID 2236179