[2026-01-10 02:15:05,890][  INFO][PID:2362195] Marking build as starting
[2026-01-10 02:15:06,131][  INFO][PID:2362195] Checking that /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/repodata/repomd.xml exists
[2026-01-10 02:15:06,188][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:06,203][  INFO][PID:2362195] VM allocation process starts
[2026-01-10 02:15:06,251][  INFO][PID:2362195] Trying to allocate VM: ResallocHost, ticket_id=744521, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-10 02:15:11,259][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:16,260][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:21,265][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:26,271][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:31,275][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:36,277][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:41,280][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:46,281][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:51,283][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:15:56,288][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:01,291][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:06,292][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:11,294][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:16,296][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:21,297][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:26,301][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:31,302][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:36,304][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:41,308][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:46,311][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:51,319][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:16:56,321][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:01,322][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:06,323][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:11,325][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:16,327][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:21,329][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:24,466][  INFO][PID:2362195] Allocated host ResallocHost, ticket_id=744521, hostname=2620:52:6:1161:dead:beef:cafe:c14e, name=vmhost_p09_03_prod_01076497_20260110_020921, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-10 02:17:24,468][  INFO][PID:2362195] Allocating ssh connection to builder
[2026-01-10 02:17:24,469][  INFO][PID:2362195] Checking that builder machine is OK
[2026-01-10 02:17:25,342][  INFO][PID:2362195] Running remote command: copr-builder-ready rhel+epel-10-ppc64le
[2026-01-10 02:17:25,864][  INFO][PID:2362195] Red Hat subscription not needed for rhel+epel-10-ppc64le
Builder is ready to be used

[2026-01-10 02:17:25,865][  INFO][PID:2362195] Filling build.info file with builder info
[2026-01-10 02:17:25,872][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:25,876][  INFO][PID:2362195] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@OpenVPN",
            "project_name": "openvpn-git",
            "project_dirname": "openvpn-git",
            "submitter": "gitlab.com:flichtenheld",
            "ended_on": null,
            "started_on": 1768011445.8750782,
            "submitted_on": null,
            "status": 3,
            "chroot": "rhel+epel-10-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9993506-rhel+epel-10-ppc64le",
            "build_id": 9993506,
            "package_name": "openvpn",
            "package_version": "2.7-20260109git50a6b66",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@OpenVPN/openvpn-git/openvpn",
            "git_hash": "8eb7ab32c72b587372a2ee632855331f4ba0b0bb",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@OpenVPN/openvpn-git--gitlab.com:flichtenheld",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@OpenVPN/openvpn-git",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-git",
            "result_dir": "09993506-openvpn",
            "built_packages": "",
            "id": 9993506,
            "mockchain_macros": {
                "copr_username": "@OpenVPN",
                "copr_projectname": "openvpn-git",
                "vendor": "Fedora Project COPR (@OpenVPN/openvpn-git)"
            }
        }
    ]
}
[2026-01-10 02:17:25,993][  INFO][PID:2362195] Sending fedora-messaging bus message in build.start
[2026-01-10 02:17:26,741][  INFO][PID:2362195] Sending fedora-messaging bus message in chroot.start
[2026-01-10 02:17:26,771][  INFO][PID:2362195] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9993506-rhel+epel-10-ppc64le --chroot rhel+epel-10-ppc64le --detached
[2026-01-10 02:17:27,770][  INFO][PID:2362195] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14e' (ED25519) to the list of known hosts.


[2026-01-10 02:17:27,772][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 1
[2026-01-10 02:17:27,774][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14e copr-rpmbuild-log
[2026-01-10 02:17:32,789][  INFO][PID:2362195] Periodic builder liveness probe: alive
[2026-01-10 02:17:32,790][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:37,795][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:42,798][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:47,801][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:52,804][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:17:57,812][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:02,814][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:07,817][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:12,821][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:17,828][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:22,837][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:27,839][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:32,843][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:37,844][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:42,847][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:47,851][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:52,853][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:18:57,854][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:02,857][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:07,858][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:12,860][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:17,861][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:22,862][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:27,864][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:32,867][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:37,868][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:42,869][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:47,870][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:52,871][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:19:57,872][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:02,873][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:07,874][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:12,875][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:17,876][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:22,878][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:27,880][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:32,881][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:37,882][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:42,883][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:47,884][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:52,885][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:20:57,886][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:02,887][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:07,889][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:12,890][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:17,891][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:22,892][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:27,893][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:32,895][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:37,897][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:42,898][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:47,899][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:52,901][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:21:57,902][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:02,903][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:07,904][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:12,907][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:17,911][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:19,370][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-10 02:22:22,912][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:27,913][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:29,372][WARNING][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-10 02:22:32,915][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:37,917][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:39,376][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 2
[2026-01-10 02:22:39,377][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14e copr-rpmbuild-log
[2026-01-10 02:22:42,921][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:47,923][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:52,924][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:22:57,927][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:02,929][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:07,930][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:12,932][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:17,933][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:22,934][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:27,936][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:32,937][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:37,938][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:39,446][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-10 02:23:42,939][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:47,940][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:49,454][WARNING][PID:2362195] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-10 02:23:52,941][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:57,942][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:23:59,455][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 3
[2026-01-10 02:23:59,456][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14e copr-rpmbuild-log
[2026-01-10 02:24:02,943][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:07,944][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:12,945][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:17,946][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:22,947][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:27,948][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:32,949][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:37,950][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:42,951][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:47,952][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:52,953][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:57,954][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:24:59,527][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-10 02:25:02,956][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:07,957][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:09,527][WARNING][PID:2362195] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-10 02:25:12,958][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:17,959][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:19,528][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 4
[2026-01-10 02:25:19,529][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14e copr-rpmbuild-log
[2026-01-10 02:25:22,964][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:27,965][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:32,966][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:37,968][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:42,969][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:47,972][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:52,979][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:25:57,983][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:02,984][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:07,986][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:12,987][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:17,990][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:19,593][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-10 02:26:22,991][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:27,992][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:29,594][ ERROR][PID:2362195] Giving up for unstable SSH, failures: 1768011749.3726702, 1768011829.454529, 1768011909.5278695, 1768011989.594694
[2026-01-10 02:26:29,597][ ERROR][PID:2362195] Re-try request for task on 'ResallocHost, ticket_id=744521, hostname=2620:52:6:1161:dead:beef:cafe:c14e, name=vmhost_p09_03_prod_01076497_20260110_020921, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-10 02:26:29,597][  INFO][PID:2362195] Releasing VM back to pool
[2026-01-10 02:26:29,612][  INFO][PID:2362195] Retry #1 (on other host)
[2026-01-10 02:26:29,612][  INFO][PID:2362195] Checking that /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/repodata/repomd.xml exists
[2026-01-10 02:26:29,613][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:26:29,614][  INFO][PID:2362195] VM allocation process starts
[2026-01-10 02:26:29,627][  INFO][PID:2362195] Trying to allocate VM: ResallocHost, ticket_id=744881, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-10 02:26:32,640][  INFO][PID:2362195] Allocated host ResallocHost, ticket_id=744881, hostname=2620:52:6:1161:dead:beef:cafe:c152, name=vmhost_p09_03_prod_01076672_20260110_021918, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-10 02:26:32,640][  INFO][PID:2362195] Allocating ssh connection to builder
[2026-01-10 02:26:32,640][  INFO][PID:2362195] Checking that builder machine is OK
[2026-01-10 02:27:32,709][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c152 port 22: Connection timed out

[2026-01-10 02:27:42,710][ ERROR][PID:2362195] Re-try request for task on 'ResallocHost, ticket_id=744881, hostname=2620:52:6:1161:dead:beef:cafe:c152, name=vmhost_p09_03_prod_01076672_20260110_021918, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2026-01-10 02:27:42,710][  INFO][PID:2362195] Releasing VM back to pool
[2026-01-10 02:27:42,719][  INFO][PID:2362195] Retry #2 (on other host)
[2026-01-10 02:27:42,720][  INFO][PID:2362195] Checking that /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/repodata/repomd.xml exists
[2026-01-10 02:27:42,720][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:27:42,721][  INFO][PID:2362195] VM allocation process starts
[2026-01-10 02:27:42,732][  INFO][PID:2362195] Trying to allocate VM: ResallocHost, ticket_id=744898, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-10 02:27:45,750][  INFO][PID:2362195] Allocated host ResallocHost, ticket_id=744898, hostname=2620:52:6:1161:dead:beef:cafe:c18e, name=vmhost_p09_04_prod_01076686_20260110_022049, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-10 02:27:45,750][  INFO][PID:2362195] Allocating ssh connection to builder
[2026-01-10 02:27:45,751][  INFO][PID:2362195] Checking that builder machine is OK
[2026-01-10 02:27:46,704][  INFO][PID:2362195] Running remote command: copr-builder-ready rhel+epel-10-ppc64le
[2026-01-10 02:27:47,189][  INFO][PID:2362195] Red Hat subscription not needed for rhel+epel-10-ppc64le
Builder is ready to be used

[2026-01-10 02:27:47,190][  INFO][PID:2362195] Filling build.info file with builder info
[2026-01-10 02:27:47,221][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:27:47,223][  INFO][PID:2362195] Not re-notifying FE and msg buses for the new host.
[2026-01-10 02:27:47,223][  INFO][PID:2362195] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9993506-rhel+epel-10-ppc64le --chroot rhel+epel-10-ppc64le --detached
[2026-01-10 02:27:48,158][  INFO][PID:2362195] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18e' (ED25519) to the list of known hosts.


[2026-01-10 02:27:48,159][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 1
[2026-01-10 02:27:48,160][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log
[2026-01-10 02:27:53,164][  INFO][PID:2362195] Periodic builder liveness probe: alive
[2026-01-10 02:27:53,165][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:27:58,166][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:03,167][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:08,168][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:13,170][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:18,171][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:23,172][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:28,173][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:33,174][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:38,175][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:43,176][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:48,176][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:48,228][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-10 02:28:53,177][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:58,179][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:28:58,241][WARNING][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-10 02:29:03,180][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:08,181][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:08,241][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 2
[2026-01-10 02:29:08,243][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log
[2026-01-10 02:29:13,182][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:18,183][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:23,184][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:28,184][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:33,186][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:38,187][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:43,195][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:48,196][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:53,197][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:29:58,198][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:03,199][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:08,200][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:08,309][ ERROR][PID:2362195] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-10 02:30:13,201][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:18,202][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:18,318][WARNING][PID:2362195] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-10 02:30:23,203][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:28,204][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:28,319][  INFO][PID:2362195] Downloading the builder-live.log file, attempt 3
[2026-01-10 02:30:28,320][  INFO][PID:2362195] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log
[2026-01-10 02:30:33,205][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:38,206][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:43,207][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:48,208][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:53,231][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:30:58,235][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:03,237][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:08,238][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:13,239][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:18,240][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:23,241][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:28,242][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:33,243][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:38,244][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:43,245][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:48,246][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:53,247][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:31:58,248][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:03,249][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:08,250][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:13,251][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:18,252][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:23,253][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:28,254][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:33,255][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:38,256][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:43,257][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:48,258][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:53,259][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:32:58,261][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:03,262][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:08,263][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:13,416][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:18,417][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:23,419][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:28,424][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:33,428][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:38,469][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:43,529][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:48,534][  INFO][PID:2362195] Checking for cancel request
[2026-01-10 02:33:51,951][  INFO][PID:2362195] Downloading results from builder
[2026-01-10 02:33:51,952][  INFO][PID:2362195] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18e]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn started
[2026-01-10 02:33:51,953][  INFO][PID:2362195] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18e]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/ &> /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/build-09993506.rsync.log
[2026-01-10 02:33:53,002][  INFO][PID:2362195] rsyncing finished.
[2026-01-10 02:33:53,003][  INFO][PID:2362195] Releasing VM back to pool
[2026-01-10 02:33:53,013][  INFO][PID:2362195] Searching for 'success' file in resultdir
[2026-01-10 02:33:53,014][ ERROR][PID:2362195] Build failed: Backend process error: No success file => build failure
[2026-01-10 02:33:53,014][  INFO][PID:2362195] Finished build: id=9993506 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@OpenVPN/openvpn-git chroot=rhel+epel-10-ppc64le 
[2026-01-10 02:33:53,016][ ERROR][PID:2362195] Backend process error: No success file => build failure
[2026-01-10 02:33:53,016][  INFO][PID:2362195] Worker failed build, took 365.7936396598816
[2026-01-10 02:33:53,017][  INFO][PID:2362195] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@OpenVPN",
            "project_name": "openvpn-git",
            "project_dirname": "openvpn-git",
            "submitter": "gitlab.com:flichtenheld",
            "ended_on": 1768012433.0168908,
            "started_on": 1768012067.223251,
            "submitted_on": null,
            "status": 0,
            "chroot": "rhel+epel-10-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9993506-rhel+epel-10-ppc64le",
            "build_id": 9993506,
            "package_name": "openvpn",
            "package_version": "2.7-20260109git50a6b66",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@OpenVPN/openvpn-git/openvpn",
            "git_hash": "8eb7ab32c72b587372a2ee632855331f4ba0b0bb",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@OpenVPN/openvpn-git--gitlab.com:flichtenheld",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@OpenVPN/openvpn-git",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@OpenVPN/openvpn-git",
            "result_dir": "09993506-openvpn",
            "built_packages": "",
            "id": 9993506,
            "mockchain_macros": {
                "copr_username": "@OpenVPN",
                "copr_projectname": "openvpn-git",
                "vendor": "Fedora Project COPR (@OpenVPN/openvpn-git)"
            }
        }
    ]
}
[2026-01-10 02:33:53,108][  INFO][PID:2362195] Sending fedora-messaging bus message in build.end
[2026-01-10 02:33:53,135][  INFO][PID:2362195] Compressing /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/builder-live.log by gzip
[2026-01-10 02:33:53,136][  INFO][PID:2362195] Running command 'gzip /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/builder-live.log' as PID 2487264
[2026-01-10 02:33:53,186][  INFO][PID:2362195] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/builder-live.log)
[2026-01-10 02:33:53,187][  INFO][PID:2362195] Compressing /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/backend.log by gzip
[2026-01-10 02:33:53,188][  INFO][PID:2362195] Running command 'gzip /var/lib/copr/public_html/results/@OpenVPN/openvpn-git/rhel+epel-10-ppc64le/09993506-openvpn/backend.log' as PID 2487265