[2025-12-27 20:39:46,725][  INFO][PID:3187208] Marking build as starting
[2025-12-27 20:39:46,826][  INFO][PID:3187208] Checking that /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/repodata/repomd.xml exists
[2025-12-27 20:39:46,846][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:39:46,847][  INFO][PID:3187208] VM allocation process starts
[2025-12-27 20:39:46,861][  INFO][PID:3187208] Trying to allocate VM: ResallocHost, ticket_id=538245, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-27 20:39:51,862][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:39:56,863][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:01,864][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:06,865][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:11,866][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:16,867][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:21,868][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:26,869][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:31,870][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:36,872][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:41,873][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:46,874][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:51,877][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:40:56,878][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:01,879][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:06,880][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:11,881][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:16,882][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:21,883][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:26,884][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:31,885][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:36,886][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:41,887][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:46,888][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:51,889][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:41:56,890][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:01,891][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:06,892][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:11,893][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:16,894][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:21,895][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:26,896][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:31,898][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:36,899][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:41,899][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:46,900][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:51,901][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:42:56,902][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:01,903][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:06,904][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:11,905][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:16,906][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:21,907][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:26,908][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:31,909][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:36,910][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:41,911][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:46,912][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:51,913][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:43:56,914][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:01,914][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:05,016][  INFO][PID:3187208] Allocated host ResallocHost, ticket_id=538245, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00793528_20251227_203922, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-27 20:44:05,017][  INFO][PID:3187208] Allocating ssh connection to builder
[2025-12-27 20:44:05,017][  INFO][PID:3187208] Checking that builder machine is OK
[2025-12-27 20:44:05,913][  INFO][PID:3187208] Running remote command: copr-builder-ready epel-9-ppc64le
[2025-12-27 20:44:06,769][  INFO][PID:3187208] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-12-27 20:44:06,770][  INFO][PID:3187208] Filling build.info file with builder info
[2025-12-27 20:44:06,770][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:06,771][  INFO][PID:3187208] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "sassam",
            "project_name": "dr14_tmeter",
            "project_dirname": "dr14_tmeter",
            "submitter": "sassam",
            "ended_on": null,
            "started_on": 1766868246.7710605,
            "submitted_on": null,
            "status": 3,
            "chroot": "epel-9-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9953825-epel-9-ppc64le",
            "build_id": 9953825,
            "package_name": "dr14_t.meter",
            "package_version": "1.0.16-10",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/sassam/dr14_tmeter/dr14_t.meter",
            "git_hash": "6e281bc0bcbb9753f323af6fe9cb96b01816cdb5",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "sassam/dr14_tmeter--sassam",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/sassam/dr14_tmeter/epel-9-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/sassam/dr14_tmeter",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/sassam/dr14_tmeter",
            "result_dir": "09953825-dr14_t.meter",
            "built_packages": "",
            "id": 9953825,
            "mockchain_macros": {
                "copr_username": "sassam",
                "copr_projectname": "dr14_tmeter",
                "vendor": "Fedora Project COPR (sassam/dr14_tmeter)"
            }
        }
    ]
}
[2025-12-27 20:44:06,888][  INFO][PID:3187208] Sending fedora-messaging bus message in build.start
[2025-12-27 20:44:07,358][  INFO][PID:3187208] Sending fedora-messaging bus message in chroot.start
[2025-12-27 20:44:07,383][  INFO][PID:3187208] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9953825-epel-9-ppc64le --chroot epel-9-ppc64le --detached
[2025-12-27 20:44:08,349][  INFO][PID:3187208] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18c' (ED25519) to the list of known hosts.


[2025-12-27 20:44:08,350][  INFO][PID:3187208] Downloading the builder-live.log file, attempt 1
[2025-12-27 20:44:08,352][  INFO][PID:3187208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18c copr-rpmbuild-log
[2025-12-27 20:44:13,355][  INFO][PID:3187208] Periodic builder liveness probe: alive
[2025-12-27 20:44:13,356][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:18,356][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:23,357][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:28,358][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:33,359][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:38,360][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:43,361][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:48,362][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:53,363][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:44:58,364][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:03,365][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:08,366][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:13,367][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:18,368][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:23,369][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:28,370][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:33,371][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:38,372][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:43,373][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:48,373][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:53,374][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:45:58,375][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:03,376][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:08,377][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:13,378][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:18,379][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:23,380][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:28,381][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:33,382][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:38,383][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:43,384][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:48,385][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:53,386][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:46:58,387][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:47:03,388][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:47:08,389][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:47:13,390][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:47:18,391][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:47:20,418][  INFO][PID:3187208] Downloading results from builder
[2025-12-27 20:47:20,419][  INFO][PID:3187208] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18c]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter started
[2025-12-27 20:47:20,419][  INFO][PID:3187208] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18c]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/ &> /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/build-09953825.rsync.log
[2025-12-27 20:48:20,484][  INFO][PID:3187208] rsyncing finished.
[2025-12-27 20:48:20,485][ ERROR][PID:3187208] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-27 20:48:20,485][ ERROR][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-27 20:48:30,485][  INFO][PID:3187208] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18c]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter started
[2025-12-27 20:48:30,486][  INFO][PID:3187208] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18c]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/ &> /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/build-09953825.rsync.log
[2025-12-27 20:49:30,582][  INFO][PID:3187208] rsyncing finished.
[2025-12-27 20:49:30,583][ ERROR][PID:3187208] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-27 20:49:30,583][ ERROR][PID:3187208] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-27 20:49:40,584][  INFO][PID:3187208] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18c]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter started
[2025-12-27 20:49:40,585][  INFO][PID:3187208] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18c]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/ &> /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/build-09953825.rsync.log
[2025-12-27 20:50:40,640][  INFO][PID:3187208] rsyncing finished.
[2025-12-27 20:50:40,657][ ERROR][PID:3187208] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-27 20:50:40,657][ ERROR][PID:3187208] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-27 20:50:50,658][ ERROR][PID:3187208] Re-try request for task on 'ResallocHost, ticket_id=538245, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00793528_20251227_203922, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts
[2025-12-27 20:50:50,658][  INFO][PID:3187208] Releasing VM back to pool
[2025-12-27 20:50:50,667][  INFO][PID:3187208] Retry #1 (on other host)
[2025-12-27 20:50:50,668][  INFO][PID:3187208] Checking that /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/repodata/repomd.xml exists
[2025-12-27 20:50:50,669][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:50:50,669][  INFO][PID:3187208] VM allocation process starts
[2025-12-27 20:50:50,733][  INFO][PID:3187208] Trying to allocate VM: ResallocHost, ticket_id=538283, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-27 20:50:53,781][  INFO][PID:3187208] Allocated host ResallocHost, ticket_id=538283, hostname=2620:52:6:1161:dead:beef:cafe:c155, name=vmhost_p09_03_prod_00793515_20251227_203843, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-27 20:50:53,782][  INFO][PID:3187208] Allocating ssh connection to builder
[2025-12-27 20:50:53,782][  INFO][PID:3187208] Checking that builder machine is OK
[2025-12-27 20:50:54,248][  INFO][PID:3187208] Running remote command: copr-builder-ready epel-9-ppc64le
[2025-12-27 20:50:54,735][  INFO][PID:3187208] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-12-27 20:50:54,979][  INFO][PID:3187208] Filling build.info file with builder info
[2025-12-27 20:50:55,413][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:50:55,414][  INFO][PID:3187208] Not re-notifying FE and msg buses for the new host.
[2025-12-27 20:50:55,415][  INFO][PID:3187208] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9953825-epel-9-ppc64le --chroot epel-9-ppc64le --detached
[2025-12-27 20:51:55,425][ ERROR][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c155 port 22: Connection timed out

[2025-12-27 20:52:05,426][ ERROR][PID:3187208] Re-try request for task on 'ResallocHost, ticket_id=538283, hostname=2620:52:6:1161:dead:beef:cafe:c155, name=vmhost_p09_03_prod_00793515_20251227_203843, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-27 20:52:05,426][  INFO][PID:3187208] Releasing VM back to pool
[2025-12-27 20:52:05,435][  INFO][PID:3187208] Retry #2 (on other host)
[2025-12-27 20:52:05,436][  INFO][PID:3187208] Checking that /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/repodata/repomd.xml exists
[2025-12-27 20:52:05,436][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:05,437][  INFO][PID:3187208] VM allocation process starts
[2025-12-27 20:52:05,479][  INFO][PID:3187208] Trying to allocate VM: ResallocHost, ticket_id=538285, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-27 20:52:08,513][  INFO][PID:3187208] Allocated host ResallocHost, ticket_id=538285, hostname=2620:52:6:1161:dead:beef:cafe:c194, name=vmhost_p09_04_prod_00793545_20251227_204038, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-27 20:52:08,513][  INFO][PID:3187208] Allocating ssh connection to builder
[2025-12-27 20:52:08,514][  INFO][PID:3187208] Checking that builder machine is OK
[2025-12-27 20:52:09,433][  INFO][PID:3187208] Running remote command: copr-builder-ready epel-9-ppc64le
[2025-12-27 20:52:09,946][  INFO][PID:3187208] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-12-27 20:52:09,947][  INFO][PID:3187208] Filling build.info file with builder info
[2025-12-27 20:52:09,954][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:09,955][  INFO][PID:3187208] Not re-notifying FE and msg buses for the new host.
[2025-12-27 20:52:09,955][  INFO][PID:3187208] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9953825-epel-9-ppc64le --chroot epel-9-ppc64le --detached
[2025-12-27 20:52:10,916][  INFO][PID:3187208] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c194' (ED25519) to the list of known hosts.


[2025-12-27 20:52:10,917][  INFO][PID:3187208] Downloading the builder-live.log file, attempt 1
[2025-12-27 20:52:10,918][  INFO][PID:3187208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c194 copr-rpmbuild-log
[2025-12-27 20:52:15,917][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:20,918][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:25,919][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:30,920][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:35,920][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:40,921][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:45,922][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:50,923][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:52:55,924][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:00,925][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:05,926][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:10,927][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:10,985][ ERROR][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-27 20:53:15,928][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:20,929][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:20,993][WARNING][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-27 20:53:25,930][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:30,931][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:30,994][  INFO][PID:3187208] Downloading the builder-live.log file, attempt 2
[2025-12-27 20:53:30,996][  INFO][PID:3187208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c194 copr-rpmbuild-log
[2025-12-27 20:53:35,932][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:40,932][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:45,933][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:50,934][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:53:55,935][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:00,936][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:05,937][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:10,938][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:15,943][  INFO][PID:3187208] Periodic builder liveness probe: alive
[2025-12-27 20:54:15,944][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:20,945][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:25,945][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:30,946][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:31,383][ ERROR][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-27 20:54:35,947][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:40,986][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:41,407][WARNING][PID:3187208] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-27 20:54:46,072][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:51,073][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:54:51,407][  INFO][PID:3187208] Downloading the builder-live.log file, attempt 3
[2025-12-27 20:54:51,408][  INFO][PID:3187208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c194 copr-rpmbuild-log
[2025-12-27 20:54:56,281][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:01,282][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:06,283][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:11,283][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:16,400][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:21,401][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:26,403][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:31,404][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:36,514][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:41,516][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:46,602][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:51,475][ ERROR][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-27 20:55:51,603][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:55:56,648][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:01,650][WARNING][PID:3187208] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-27 20:56:01,720][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:06,721][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:11,651][  INFO][PID:3187208] Downloading the builder-live.log file, attempt 4
[2025-12-27 20:56:11,722][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:12,040][  INFO][PID:3187208] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c194 copr-rpmbuild-log
[2025-12-27 20:56:16,722][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:22,219][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:27,220][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:32,221][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:37,221][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:42,222][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:47,223][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:52,224][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:56:57,225][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:57:02,226][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:57:07,227][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:57:12,104][ ERROR][PID:3187208] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-27 20:57:12,228][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:57:17,229][  INFO][PID:3187208] Checking for cancel request
[2025-12-27 20:57:22,136][ ERROR][PID:3187208] Giving up for unstable SSH, failures: 1766868800.9937057, 1766868881.407074, 1766868961.6508636, 1766869042.1363933
[2025-12-27 20:57:22,137][ ERROR][PID:3187208] Re-try request for task on 'ResallocHost, ticket_id=538285, hostname=2620:52:6:1161:dead:beef:cafe:c194, name=vmhost_p09_04_prod_00793545_20251227_204038, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-27 20:57:22,137][  INFO][PID:3187208] Releasing VM back to pool
[2025-12-27 20:57:22,146][ ERROR][PID:3187208] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c155', '2620:52:6:1161:dead:beef:cafe:c194', '2620:52:6:1161:dead:beef:cafe:c18c'}
[2025-12-27 20:57:22,147][WARNING][PID:3187208] Switching not-finished job state to 'failed'
[2025-12-27 20:57:22,147][  INFO][PID:3187208] Worker failed build, took 312.1918396949768
[2025-12-27 20:57:22,147][  INFO][PID:3187208] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "sassam",
            "project_name": "dr14_tmeter",
            "project_dirname": "dr14_tmeter",
            "submitter": "sassam",
            "ended_on": 1766869042.147037,
            "started_on": 1766868729.9551973,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-9-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9953825-epel-9-ppc64le",
            "build_id": 9953825,
            "package_name": "dr14_t.meter",
            "package_version": "1.0.16-10",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/sassam/dr14_tmeter/dr14_t.meter",
            "git_hash": "6e281bc0bcbb9753f323af6fe9cb96b01816cdb5",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "sassam/dr14_tmeter--sassam",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/sassam/dr14_tmeter/epel-9-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/sassam/dr14_tmeter",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/sassam/dr14_tmeter",
            "result_dir": "09953825-dr14_t.meter",
            "built_packages": "",
            "id": 9953825,
            "mockchain_macros": {
                "copr_username": "sassam",
                "copr_projectname": "dr14_tmeter",
                "vendor": "Fedora Project COPR (sassam/dr14_tmeter)"
            }
        }
    ]
}
[2025-12-27 20:57:22,241][  INFO][PID:3187208] Sending fedora-messaging bus message in build.end
[2025-12-27 20:57:22,406][  INFO][PID:3187208] Compressing /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/builder-live.log by gzip
[2025-12-27 20:57:22,407][  INFO][PID:3187208] Running command 'gzip /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/builder-live.log' as PID 3240621
[2025-12-27 20:57:22,484][  INFO][PID:3187208] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/builder-live.log)
[2025-12-27 20:57:22,485][  INFO][PID:3187208] Compressing /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/backend.log by gzip
[2025-12-27 20:57:22,486][  INFO][PID:3187208] Running command 'gzip /var/lib/copr/public_html/results/sassam/dr14_tmeter/epel-9-ppc64le/09953825-dr14_t.meter/backend.log' as PID 3240623