[2025-07-15 22:09:35,097][  INFO][PID:2178097] Marking build as starting
[2025-07-15 22:09:35,121][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:09:35,122][  INFO][PID:2178097] VM allocation process starts
[2025-07-15 22:09:35,136][  INFO][PID:2178097] Trying to allocate VM: ResallocHost, ticket_id=3947012, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-15 22:09:38,149][  INFO][PID:2178097] Allocated host ResallocHost, ticket_id=3947012, hostname=2620:52:3:1:dead:beef:cafe:c29a, name=copr_p09_01_prod_05436372_20250715_214225, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-15 22:09:38,150][  INFO][PID:2178097] Allocating ssh connection to builder
[2025-07-15 22:09:38,150][  INFO][PID:2178097] Checking that builder machine is OK
[2025-07-15 22:09:39,109][  INFO][PID:2178097] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-15 22:09:39,617][  INFO][PID:2178097] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-15 22:09:39,617][  INFO][PID:2178097] Filling build.info file with builder info
[2025-07-15 22:09:39,618][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:09:39,619][  INFO][PID:2178097] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "lecris",
            "project_name": "cmake-drop_vars",
            "project_dirname": "cmake-drop_vars",
            "submitter": "lecris",
            "ended_on": null,
            "started_on": 1752617379.6193826,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9277232-fedora-rawhide-ppc64le",
            "build_id": 9277232,
            "package_name": "llvm11",
            "package_version": "11.1.0-15",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lecris/cmake-drop_vars/llvm11",
            "git_hash": "4097411dca0629901f56be4ad932c80e5037ec41",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "lecris/cmake-drop_vars--lecris",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/lecris/cmake-drop_vars",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-drop_vars",
            "result_dir": "09277232-llvm11",
            "built_packages": "",
            "id": 9277232,
            "mockchain_macros": {
                "copr_username": "lecris",
                "copr_projectname": "cmake-drop_vars",
                "vendor": "Fedora Project COPR (lecris/cmake-drop_vars)"
            }
        }
    ]
}
[2025-07-15 22:09:39,645][  INFO][PID:2178097] Sending fedora-messaging bus message in build.start
[2025-07-15 22:09:40,269][  INFO][PID:2178097] Sending fedora-messaging bus message in chroot.start
[2025-07-15 22:09:40,293][  INFO][PID:2178097] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9277232-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-15 22:09:41,204][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 1
[2025-07-15 22:09:41,205][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-15 22:09:46,210][  INFO][PID:2178097] Periodic builder liveness probe: alive
[2025-07-15 22:09:46,211][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:09:51,212][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:09:56,212][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:01,214][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:06,215][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:11,216][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:16,217][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:21,218][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:26,219][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:31,220][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:36,221][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:41,222][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:46,223][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:51,224][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:10:56,225][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:01,226][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:06,228][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:11,229][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:16,230][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:21,231][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:26,232][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:31,233][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:36,234][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:41,235][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:46,236][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:51,237][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:11:56,238][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:01,240][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:06,241][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:11,242][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:16,243][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:21,244][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:26,246][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:31,247][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:36,248][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:41,249][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:46,250][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:51,251][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:12:56,252][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:01,253][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:06,254][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:11,255][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:16,257][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:21,258][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:26,259][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:31,260][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:36,261][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:41,262][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:46,263][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:51,264][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:13:56,266][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:01,267][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:06,268][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:11,269][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:16,270][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:21,271][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:26,272][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:31,273][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:36,274][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:41,275][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:46,277][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:51,278][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:14:56,279][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:01,280][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:06,281][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:11,282][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:16,283][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:21,284][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:26,285][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:31,286][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:36,287][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:41,288][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:46,289][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:51,290][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:15:56,291][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:01,292][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:06,293][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:11,294][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:16,295][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:21,296][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:26,298][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:31,298][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:36,300][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:41,301][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:46,302][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:51,303][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:16:56,304][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:01,305][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:06,306][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:11,307][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:16,308][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:21,309][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:26,310][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:31,311][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:36,312][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:41,313][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:46,314][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:51,315][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:17:56,316][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:01,317][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:06,318][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:11,319][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:16,320][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:21,321][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:26,322][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:31,323][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:36,324][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:41,325][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:46,326][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:51,327][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:18:56,329][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:01,330][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:06,331][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:11,332][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:16,333][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:21,334][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:26,335][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:31,336][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:36,337][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:41,338][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:46,347][  INFO][PID:2178097] Periodic builder liveness probe: alive
[2025-07-15 22:19:46,347][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:51,348][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:19:56,349][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:01,350][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:06,351][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:11,352][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:16,353][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:21,354][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:26,355][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:31,356][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:36,357][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:41,358][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:46,359][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:51,360][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:20:56,361][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:01,362][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:06,363][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:11,364][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:16,365][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:21,366][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:26,368][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:31,369][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:36,370][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:41,371][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:46,372][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:51,373][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:21:56,374][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:01,375][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:06,377][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:11,378][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:16,379][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:21,380][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:26,381][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:31,382][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:36,383][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:41,384][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:46,385][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:51,386][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:22:56,387][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:01,388][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:06,389][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:11,390][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:16,391][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:21,393][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:26,393][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:31,395][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:36,396][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:41,397][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:46,398][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:50,139][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:23:51,399][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:23:56,400][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:00,140][WARNING][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 22:24:01,401][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:06,401][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:10,140][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 2
[2025-07-15 22:24:10,143][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-15 22:24:11,404][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:16,405][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:21,407][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:26,407][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:31,409][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:36,410][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:41,411][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:46,412][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:51,413][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:24:56,414][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:01,415][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:06,416][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:10,210][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:25:11,417][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:16,418][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:20,213][WARNING][PID:2178097] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 22:25:21,419][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:26,420][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:30,214][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 3
[2025-07-15 22:25:30,215][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-15 22:25:31,421][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:36,422][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:41,423][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:46,424][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:51,425][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:25:56,426][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:01,427][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:06,428][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:11,429][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:16,430][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:21,431][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:26,433][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:30,283][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:26:31,434][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:36,435][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:40,286][WARNING][PID:2178097] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 22:26:41,436][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:46,437][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:50,287][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 4
[2025-07-15 22:26:50,288][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-15 22:26:51,438][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:26:56,439][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:01,440][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:06,441][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:11,442][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:16,443][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:21,444][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:26,445][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:31,446][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:36,447][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:41,448][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:46,449][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:50,356][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:27:51,450][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:27:56,452][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:00,359][ ERROR][PID:2178097] Giving up for unstable SSH, failures: 1752618240.140308, 1752618320.213805, 1752618400.2868521, 1752618480.359499
[2025-07-15 22:28:00,360][ ERROR][PID:2178097] Re-try request for task on 'ResallocHost, ticket_id=3947012, hostname=2620:52:3:1:dead:beef:cafe:c29a, name=copr_p09_01_prod_05436372_20250715_214225, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-15 22:28:00,361][  INFO][PID:2178097] Releasing VM back to pool
[2025-07-15 22:28:00,369][  INFO][PID:2178097] Retry #1 (on other host)
[2025-07-15 22:28:00,370][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:00,370][  INFO][PID:2178097] VM allocation process starts
[2025-07-15 22:28:00,426][  INFO][PID:2178097] Trying to allocate VM: ResallocHost, ticket_id=3947232, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-15 22:28:03,468][  INFO][PID:2178097] Allocated host ResallocHost, ticket_id=3947232, hostname=2620:52:3:1:dead:beef:cafe:c28c, name=copr_p09_01_prod_05436921_20250715_221600, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-15 22:28:03,469][  INFO][PID:2178097] Allocating ssh connection to builder
[2025-07-15 22:28:03,469][  INFO][PID:2178097] Checking that builder machine is OK
[2025-07-15 22:28:04,663][  INFO][PID:2178097] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-15 22:28:05,189][  INFO][PID:2178097] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-15 22:28:05,189][  INFO][PID:2178097] Filling build.info file with builder info
[2025-07-15 22:28:05,193][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:05,194][  INFO][PID:2178097] Not re-notifying FE and msg buses for the new host.
[2025-07-15 22:28:05,194][  INFO][PID:2178097] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9277232-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-15 22:28:06,475][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 1
[2025-07-15 22:28:06,476][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28c copr-rpmbuild-log
[2025-07-15 22:28:11,476][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:16,477][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:21,478][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:26,479][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:31,481][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:36,482][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:41,483][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:46,484][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:51,485][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:28:56,486][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:01,488][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:06,489][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:11,490][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:16,491][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:21,492][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:26,493][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:31,494][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:36,495][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:41,496][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:46,502][  INFO][PID:2178097] Periodic builder liveness probe: alive
[2025-07-15 22:29:46,503][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:51,504][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:29:56,505][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:01,506][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:06,507][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:11,508][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:16,510][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:21,511][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:26,512][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:31,513][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:36,514][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:41,515][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:46,516][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:51,518][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:30:56,518][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:01,520][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:06,521][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:11,522][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:16,523][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:21,524][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:26,525][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:31,526][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:36,527][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:41,528][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:46,529][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:51,530][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:31:56,531][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:01,533][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:06,534][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:11,535][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:16,536][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:21,537][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:26,538][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:31,539][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:36,540][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:41,541][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:46,543][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:51,544][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:32:56,545][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:01,546][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:06,547][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:11,548][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:16,550][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:21,551][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:26,552][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:31,553][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:36,554][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:41,555][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:46,556][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:51,557][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:33:56,558][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:01,559][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:06,560][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:11,562][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:16,563][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:21,564][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:26,565][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:31,567][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:36,569][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:41,570][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:46,571][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:51,572][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:34:56,573][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:01,574][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:06,575][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:11,576][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:16,578][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:21,579][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:26,580][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:31,581][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:36,582][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:41,583][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:46,584][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:51,585][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:35:56,586][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:01,587][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:06,589][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:11,590][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:16,591][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:21,592][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:26,593][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:31,594][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:36,595][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:41,596][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:46,597][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:51,598][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:36:56,599][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:01,600][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:06,601][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:11,603][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:16,604][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:21,605][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:26,606][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:31,607][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:36,608][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:41,609][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:46,610][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:51,611][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:37:56,613][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:01,614][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:06,615][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:11,616][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:16,617][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:21,618][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:26,619][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:31,620][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:36,621][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:41,622][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:46,624][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:51,625][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:38:56,626][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:01,627][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:06,628][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:11,629][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:16,630][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:21,631][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:26,632][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:31,633][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:36,634][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:41,635][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:46,641][  INFO][PID:2178097] Periodic builder liveness probe: alive
[2025-07-15 22:39:46,641][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:51,642][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:39:56,643][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:01,644][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:06,645][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:11,647][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:16,648][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:21,649][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:26,650][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:31,651][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:36,652][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:41,652][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:46,653][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:51,655][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:40:56,656][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:01,657][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:06,658][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:11,659][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:16,661][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:21,662][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:26,663][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:31,664][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:36,665][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:41,666][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:46,667][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:51,669][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:41:56,670][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:01,671][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:06,672][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:11,673][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:16,674][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:21,675][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:26,676][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:31,678][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:36,679][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:41,681][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:46,682][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:51,684][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:42:56,685][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:01,686][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:06,688][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:11,689][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:16,690][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:21,691][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:26,692][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:31,693][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:36,694][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:41,695][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:46,697][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:51,698][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:43:56,699][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:01,700][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:06,701][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:11,702][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:16,703][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:21,704][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:26,705][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:31,706][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:36,707][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:41,708][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:46,709][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:51,710][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:44:56,711][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:01,712][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:06,713][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:11,714][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:16,715][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:21,716][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:26,717][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:31,718][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:36,719][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:41,720][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:46,721][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:51,722][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:45:56,723][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:01,724][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:06,725][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:11,726][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:16,727][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:21,729][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:26,730][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:31,731][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:36,732][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:41,733][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:46,734][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:51,735][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:46:56,736][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:01,737][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:06,738][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:11,739][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:16,741][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:21,742][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:26,743][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:31,744][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:36,745][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:41,746][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:46,747][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:51,748][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:47:56,749][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:01,750][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:06,751][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:11,752][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:16,753][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:21,754][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:26,755][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:31,756][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:36,757][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:41,758][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:46,759][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:51,760][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:48:56,761][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:01,762][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:06,763][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:11,764][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:16,765][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:21,767][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:26,768][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:31,769][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:36,770][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:41,771][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:46,776][  INFO][PID:2178097] Periodic builder liveness probe: alive
[2025-07-15 22:49:46,777][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:51,777][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:49:56,778][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:01,779][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:06,780][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:11,781][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:16,782][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:20,139][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:50:21,783][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:26,784][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:30,140][WARNING][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 22:50:31,785][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:36,787][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:40,140][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 2
[2025-07-15 22:50:40,143][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28c copr-rpmbuild-log
[2025-07-15 22:50:41,788][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:46,789][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:51,790][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:50:56,791][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:01,792][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:06,793][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:11,794][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:16,795][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:21,796][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:26,797][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:31,798][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:36,800][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:40,206][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:51:41,801][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:46,802][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:50,208][WARNING][PID:2178097] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 22:51:51,803][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:51:56,804][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:00,209][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 3
[2025-07-15 22:52:00,210][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28c copr-rpmbuild-log
[2025-07-15 22:52:01,805][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:06,806][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:11,808][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:16,809][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:21,810][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:26,811][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:31,812][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:36,813][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:41,814][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:46,815][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:51,816][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:52:56,817][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:00,278][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:53:01,818][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:06,819][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:10,402][WARNING][PID:2178097] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 22:53:11,820][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:16,821][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:20,403][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 4
[2025-07-15 22:53:20,404][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28c copr-rpmbuild-log
[2025-07-15 22:53:21,822][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:26,824][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:31,825][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:36,826][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:41,827][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:46,828][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:51,829][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:53:56,830][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:01,831][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:06,832][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:11,832][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:16,833][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:20,458][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 22:54:21,835][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:26,836][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:30,460][ ERROR][PID:2178097] Giving up for unstable SSH, failures: 1752619830.1400185, 1752619910.208347, 1752619990.4025977, 1752620070.4605129
[2025-07-15 22:54:30,461][ ERROR][PID:2178097] Re-try request for task on 'ResallocHost, ticket_id=3947232, hostname=2620:52:3:1:dead:beef:cafe:c28c, name=copr_p09_01_prod_05436921_20250715_221600, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-15 22:54:30,461][  INFO][PID:2178097] Releasing VM back to pool
[2025-07-15 22:54:30,472][  INFO][PID:2178097] Retry #2 (on other host)
[2025-07-15 22:54:30,473][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:30,473][  INFO][PID:2178097] VM allocation process starts
[2025-07-15 22:54:30,483][  INFO][PID:2178097] Trying to allocate VM: ResallocHost, ticket_id=3947561, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-15 22:54:33,495][  INFO][PID:2178097] Allocated host ResallocHost, ticket_id=3947561, hostname=140.211.168.89, name=copr_osuosl_p09_prod_05436610_20250715_215700, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-15 22:54:33,495][  INFO][PID:2178097] Allocating ssh connection to builder
[2025-07-15 22:54:33,495][  INFO][PID:2178097] Checking that builder machine is OK
[2025-07-15 22:54:34,931][  INFO][PID:2178097] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-15 22:54:35,947][  INFO][PID:2178097] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-15 22:54:35,947][  INFO][PID:2178097] Filling build.info file with builder info
[2025-07-15 22:54:35,948][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:35,949][  INFO][PID:2178097] Not re-notifying FE and msg buses for the new host.
[2025-07-15 22:54:35,949][  INFO][PID:2178097] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9277232-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-15 22:54:37,274][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 1
[2025-07-15 22:54:37,275][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.89 copr-rpmbuild-log
[2025-07-15 22:54:42,274][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:47,276][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:52,277][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:54:57,278][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:02,279][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:07,280][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:12,281][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:17,282][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:22,283][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:27,284][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:32,285][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:37,286][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:42,287][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:47,288][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:52,290][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:55:57,291][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:02,293][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:07,294][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:12,295][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:17,296][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:22,297][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:27,298][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:32,300][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:37,301][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:42,302][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:47,303][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:52,304][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:56:57,305][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:02,306][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:07,307][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:12,308][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:17,309][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:22,310][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:27,311][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:32,312][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:37,313][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:42,314][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:47,315][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:52,316][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:57:57,317][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:02,318][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:07,319][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:12,320][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:17,321][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:22,323][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:27,324][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:32,325][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:37,326][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:42,327][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:47,328][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:52,329][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:58:57,330][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:02,331][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:07,332][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:12,333][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:17,334][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:22,336][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:27,337][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:32,338][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:37,339][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:42,340][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:47,346][  INFO][PID:2178097] Periodic builder liveness probe: alive
[2025-07-15 22:59:47,347][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:52,348][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 22:59:57,394][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:02,395][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:07,396][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:12,397][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:17,398][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:22,399][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:27,400][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:32,402][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:37,403][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:42,404][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:47,405][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:52,406][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:00:57,407][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:02,408][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:07,409][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:12,411][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:17,412][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:22,413][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:27,414][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:32,415][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:37,416][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:42,417][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:47,418][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:52,420][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:01:57,421][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:02,422][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:07,424][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:12,426][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:17,427][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:22,428][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:27,430][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:32,431][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:37,432][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:42,434][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:47,435][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:52,437][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:02:57,438][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:02,439][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:07,441][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:12,442][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:17,443][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:22,444][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:27,445][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:32,447][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:37,448][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:42,449][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:47,450][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:52,452][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:03:57,453][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:02,454][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:07,455][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:12,456][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:17,457][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:22,458][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:27,460][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:32,461][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:37,462][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:42,463][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:47,465][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:52,467][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:04:57,468][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:02,469][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:07,470][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:12,471][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:17,472][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:22,473][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:27,474][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:32,476][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:37,477][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:42,478][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:47,479][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:52,480][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:05:57,481][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:02,482][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:07,483][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:12,485][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:17,490][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:22,497][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:27,501][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:32,502][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:37,503][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:42,503][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:47,504][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:52,505][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:06:57,507][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:02,511][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:07,512][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:12,513][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:17,515][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:22,516][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:27,518][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:32,519][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:37,521][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:42,522][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:47,524][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:52,525][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:07:57,527][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:02,529][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:07,531][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:12,532][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:17,535][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:22,536][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:27,543][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:32,544][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:37,548][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:42,550][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:47,551][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:52,552][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:08:57,553][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:02,555][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:07,558][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:12,560][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:17,562][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:22,564][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:27,565][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:32,567][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:37,568][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:42,571][  INFO][PID:2178097] Checking for cancel request
[2025-07-15 23:09:47,582][  INFO][PID:2178097] Periodic builder liveness probe: dead
[2025-07-15 23:10:27,139][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 23:10:37,140][WARNING][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 23:10:47,141][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 2
[2025-07-15 23:10:47,143][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.89 copr-rpmbuild-log
[2025-07-15 23:10:47,654][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.89 port 22: Connection timed out

[2025-07-15 23:11:47,214][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 23:11:57,217][WARNING][PID:2178097] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 23:11:57,727][ ERROR][PID:2178097] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.89 port 22: Connection timed out

[2025-07-15 23:12:07,218][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 3
[2025-07-15 23:12:07,222][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.89 copr-rpmbuild-log
[2025-07-15 23:13:07,301][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 23:13:07,805][ ERROR][PID:2178097] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.89 port 22: Connection timed out

[2025-07-15 23:13:17,305][WARNING][PID:2178097] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 23:13:27,306][  INFO][PID:2178097] Downloading the builder-live.log file, attempt 4
[2025-07-15 23:13:27,311][  INFO][PID:2178097] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.89 copr-rpmbuild-log
[2025-07-15 23:14:17,880][ ERROR][PID:2178097] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.89 port 22: Connection timed out

[2025-07-15 23:14:27,378][ ERROR][PID:2178097] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 23:14:27,881][ ERROR][PID:2178097] Can't ssh to cancel build.
[2025-07-15 23:14:37,381][ ERROR][PID:2178097] Giving up for unstable SSH, failures: 1752621037.1406894, 1752621117.2173636, 1752621197.305912, 1752621277.3817272
[2025-07-15 23:14:37,383][ ERROR][PID:2178097] Re-try request for task on 'ResallocHost, ticket_id=3947561, hostname=140.211.168.89, name=copr_osuosl_p09_prod_05436610_20250715_215700, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-07-15 23:14:37,385][  INFO][PID:2178097] Releasing VM back to pool
[2025-07-15 23:14:37,404][ ERROR][PID:2178097] Backend process error: Three host tried without success: {'2620:52:3:1:dead:beef:cafe:c28c', '2620:52:3:1:dead:beef:cafe:c29a', '140.211.168.89'}
[2025-07-15 23:14:37,405][WARNING][PID:2178097] Switching not-finished job state to 'failed'
[2025-07-15 23:14:37,406][  INFO][PID:2178097] Worker failed build, took 1201.4566488265991
[2025-07-15 23:14:37,406][  INFO][PID:2178097] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "lecris",
            "project_name": "cmake-drop_vars",
            "project_dirname": "cmake-drop_vars",
            "submitter": "lecris",
            "ended_on": 1752621277.405672,
            "started_on": 1752620075.9490232,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9277232-fedora-rawhide-ppc64le",
            "build_id": 9277232,
            "package_name": "llvm11",
            "package_version": "11.1.0-15",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lecris/cmake-drop_vars/llvm11",
            "git_hash": "4097411dca0629901f56be4ad932c80e5037ec41",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "lecris/cmake-drop_vars--lecris",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/lecris/cmake-drop_vars",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-drop_vars",
            "result_dir": "09277232-llvm11",
            "built_packages": "",
            "id": 9277232,
            "mockchain_macros": {
                "copr_username": "lecris",
                "copr_projectname": "cmake-drop_vars",
                "vendor": "Fedora Project COPR (lecris/cmake-drop_vars)"
            }
        }
    ]
}
[2025-07-15 23:14:37,430][  INFO][PID:2178097] Sending fedora-messaging bus message in build.end
[2025-07-15 23:14:37,466][  INFO][PID:2178097] Compressing /var/lib/copr/public_html/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/09277232-llvm11/builder-live.log by gzip
[2025-07-15 23:14:37,467][  INFO][PID:2178097] Running command 'gzip /var/lib/copr/public_html/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/09277232-llvm11/builder-live.log' as PID 2485263
[2025-07-15 23:14:37,469][  INFO][PID:2178097] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/09277232-llvm11/builder-live.log)
[2025-07-15 23:14:37,469][  INFO][PID:2178097] Compressing /var/lib/copr/public_html/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/09277232-llvm11/backend.log by gzip
[2025-07-15 23:14:37,470][  INFO][PID:2178097] Running command 'gzip /var/lib/copr/public_html/results/lecris/cmake-drop_vars/fedora-rawhide-ppc64le/09277232-llvm11/backend.log' as PID 2485264