[2025-12-16 13:58:51,306][  INFO][PID:470163] Marking build as starting
[2025-12-16 13:58:51,438][  INFO][PID:470163] Checking that /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-16 13:58:51,451][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:58:51,453][  INFO][PID:470163] VM allocation process starts
[2025-12-16 13:58:51,473][  INFO][PID:470163] Trying to allocate VM: ResallocHost, ticket_id=370389, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 13:58:54,492][  INFO][PID:470163] Allocated host ResallocHost, ticket_id=370389, hostname=2620:52:6:1161:dead:beef:cafe:c156, name=vmhost_p09_03_prod_00598694_20251216_134825, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 13:58:54,493][  INFO][PID:470163] Allocating ssh connection to builder
[2025-12-16 13:58:54,493][  INFO][PID:470163] Checking that builder machine is OK
[2025-12-16 13:58:54,526][ ERROR][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c156 port 22: Permission denied

[2025-12-16 13:59:04,526][ ERROR][PID:470163] Re-try request for task on 'ResallocHost, ticket_id=370389, hostname=2620:52:6:1161:dead:beef:cafe:c156, name=vmhost_p09_03_prod_00598694_20251216_134825, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-16 13:59:04,527][  INFO][PID:470163] Releasing VM back to pool
[2025-12-16 13:59:04,537][  INFO][PID:470163] Retry #1 (on other host)
[2025-12-16 13:59:04,537][  INFO][PID:470163] Checking that /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-16 13:59:04,538][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:04,538][  INFO][PID:470163] VM allocation process starts
[2025-12-16 13:59:04,548][  INFO][PID:470163] Trying to allocate VM: ResallocHost, ticket_id=370402, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 13:59:07,563][  INFO][PID:470163] Allocated host ResallocHost, ticket_id=370402, hostname=2620:52:6:1161:dead:beef:cafe:c149, name=vmhost_p09_03_prod_00598772_20251216_135243, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 13:59:07,564][  INFO][PID:470163] Allocating ssh connection to builder
[2025-12-16 13:59:07,564][  INFO][PID:470163] Checking that builder machine is OK
[2025-12-16 13:59:08,470][  INFO][PID:470163] Running remote command: copr-builder-ready fedora-42-ppc64le
[2025-12-16 13:59:08,952][  INFO][PID:470163] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2025-12-16 13:59:08,952][  INFO][PID:470163] Filling build.info file with builder info
[2025-12-16 13:59:08,953][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:08,953][  INFO][PID:470163] Not re-notifying FE and msg buses for the new host.
[2025-12-16 13:59:08,954][  INFO][PID:470163] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9915112-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2025-12-16 13:59:09,946][  INFO][PID:470163] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c149' (ED25519) to the list of known hosts.


[2025-12-16 13:59:09,947][  INFO][PID:470163] Downloading the builder-live.log file, attempt 1
[2025-12-16 13:59:09,949][  INFO][PID:470163] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c149 copr-rpmbuild-log
[2025-12-16 13:59:14,953][  INFO][PID:470163] Periodic builder liveness probe: alive
[2025-12-16 13:59:14,954][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:19,955][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:24,994][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:29,995][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:35,110][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:40,111][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:45,112][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:50,113][  INFO][PID:470163] Checking for cancel request
[2025-12-16 13:59:55,130][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:00,131][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:05,132][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:10,132][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:15,157][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:20,300][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:25,301][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:30,302][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:35,302][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:40,304][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:45,305][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:50,306][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:00:55,307][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:00,308][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:05,309][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:10,310][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:15,311][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:20,318][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:25,336][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:30,337][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:35,338][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:40,339][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:45,340][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:50,341][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:01:55,342][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:00,346][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:05,349][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:10,354][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:15,357][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:20,358][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:25,360][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:30,367][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:35,373][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:40,374][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:45,376][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:50,377][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:02:55,378][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:00,390][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:05,393][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:10,394][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:15,396][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:20,398][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:25,399][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:30,401][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:35,520][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:40,521][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:45,545][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:50,547][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:03:55,548][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:00,549][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:05,558][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:10,559][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:15,560][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:20,560][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:25,561][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:30,562][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:33,443][ ERROR][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-16 14:04:35,563][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:40,564][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:43,474][WARNING][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-16 14:04:45,565][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:50,566][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:04:53,486][  INFO][PID:470163] Downloading the builder-live.log file, attempt 2
[2025-12-16 14:04:53,613][  INFO][PID:470163] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c149 copr-rpmbuild-log
[2025-12-16 14:04:55,567][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:00,568][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:05,569][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:10,570][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:15,571][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:20,572][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:25,573][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:30,574][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:35,575][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:40,576][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:45,577][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:50,578][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:05:53,664][ ERROR][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-16 14:05:55,579][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:00,581][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:03,682][WARNING][PID:470163] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-16 14:06:05,584][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:10,586][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:13,683][  INFO][PID:470163] Downloading the builder-live.log file, attempt 3
[2025-12-16 14:06:13,685][  INFO][PID:470163] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c149 copr-rpmbuild-log
[2025-12-16 14:06:15,590][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:20,591][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:25,593][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:30,594][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:35,595][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:40,596][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:45,597][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:50,598][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:06:55,599][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:00,600][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:05,601][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:10,602][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:13,756][ ERROR][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-16 14:07:15,604][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:20,607][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:23,775][WARNING][PID:470163] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-16 14:07:25,608][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:30,609][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:33,776][  INFO][PID:470163] Downloading the builder-live.log file, attempt 4
[2025-12-16 14:07:33,777][  INFO][PID:470163] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c149 copr-rpmbuild-log
[2025-12-16 14:07:35,610][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:40,612][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:45,613][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:50,616][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:07:55,617][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:00,621][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:05,622][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:10,623][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:15,630][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:20,643][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:25,646][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:30,648][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:33,845][ ERROR][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-16 14:08:35,650][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:40,652][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:43,920][ ERROR][PID:470163] Giving up for unstable SSH, failures: 1765893883.4676707, 1765893963.6827009, 1765894043.77519, 1765894123.8580208
[2025-12-16 14:08:43,960][ ERROR][PID:470163] Re-try request for task on 'ResallocHost, ticket_id=370402, hostname=2620:52:6:1161:dead:beef:cafe:c149, name=vmhost_p09_03_prod_00598772_20251216_135243, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-16 14:08:43,971][  INFO][PID:470163] Releasing VM back to pool
[2025-12-16 14:08:44,181][  INFO][PID:470163] Retry #2 (on other host)
[2025-12-16 14:08:44,204][  INFO][PID:470163] Checking that /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-16 14:08:44,216][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:44,219][  INFO][PID:470163] VM allocation process starts
[2025-12-16 14:08:44,289][  INFO][PID:470163] Trying to allocate VM: ResallocHost, ticket_id=370574, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 14:08:49,350][  INFO][PID:470163] Checking for cancel request
[2025-12-16 14:08:50,537][  INFO][PID:470163] Allocated host ResallocHost, ticket_id=370574, hostname=2620:52:6:1161:dead:beef:cafe:c14a, name=vmhost_p09_03_prod_00598838_20251216_135634, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 14:08:50,544][  INFO][PID:470163] Allocating ssh connection to builder
[2025-12-16 14:08:50,549][  INFO][PID:470163] Checking that builder machine is OK
[2025-12-16 14:09:50,664][ ERROR][PID:470163] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14a port 22: Connection timed out

[2025-12-16 14:10:00,673][ ERROR][PID:470163] Re-try request for task on 'ResallocHost, ticket_id=370574, hostname=2620:52:6:1161:dead:beef:cafe:c14a, name=vmhost_p09_03_prod_00598838_20251216_135634, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-16 14:10:00,675][  INFO][PID:470163] Releasing VM back to pool
[2025-12-16 14:10:00,695][ ERROR][PID:470163] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c156', '2620:52:6:1161:dead:beef:cafe:c14a', '2620:52:6:1161:dead:beef:cafe:c149'}
[2025-12-16 14:10:00,706][WARNING][PID:470163] Switching not-finished job state to 'failed'
[2025-12-16 14:10:00,708][  INFO][PID:470163] Worker failed build, took 651.7442212104797
[2025-12-16 14:10:00,727][  INFO][PID:470163] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@mock",
            "project_name": "mock-stable",
            "project_dirname": "mock-stable",
            "submitter": "https://api.github.com/users/nikromen",
            "ended_on": 1765894200.6978676,
            "started_on": 1765893548.9536464,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9915112-fedora-42-ppc64le",
            "build_id": 9915112,
            "package_name": "mock-core-configs",
            "package_version": "43.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@mock/mock-stable/mock-core-configs",
            "git_hash": "5c6b7570e290e5fa0122017c50901527ef622f02",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@mock/mock-stable--https://api.github.com/users/nikromen",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@mock/mock-stable/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "bootstrap": "on",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@mock/mock-stable",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@mock/mock-stable",
            "result_dir": "09915112-mock-core-configs",
            "built_packages": "",
            "id": 9915112,
            "mockchain_macros": {
                "copr_username": "@mock",
                "copr_projectname": "mock-stable",
                "vendor": "Fedora Project COPR (@mock/mock-stable)"
            }
        }
    ]
}
[2025-12-16 14:10:01,534][  INFO][PID:470163] Sending fedora-messaging bus message in build.end
[2025-12-16 14:10:05,187][  INFO][PID:470163] Compressing /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/09915112-mock-core-configs/builder-live.log by gzip
[2025-12-16 14:10:05,189][  INFO][PID:470163] Running command 'gzip /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/09915112-mock-core-configs/builder-live.log' as PID 557916
[2025-12-16 14:10:05,270][  INFO][PID:470163] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/09915112-mock-core-configs/builder-live.log)
[2025-12-16 14:10:05,271][  INFO][PID:470163] Compressing /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/09915112-mock-core-configs/backend.log by gzip
[2025-12-16 14:10:05,273][  INFO][PID:470163] Running command 'gzip /var/lib/copr/public_html/results/@mock/mock-stable/fedora-42-ppc64le/09915112-mock-core-configs/backend.log' as PID 557926