[2025-12-18 01:16:51,980][  INFO][PID:1522006] Marking build as starting
[2025-12-18 01:16:52,211][  INFO][PID:1522006] Checking that /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/repodata/repomd.xml exists
[2025-12-18 01:16:52,212][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:16:52,213][  INFO][PID:1522006] VM allocation process starts
[2025-12-18 01:16:52,228][  INFO][PID:1522006] Trying to allocate VM: ResallocHost, ticket_id=415264, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-18 01:16:55,246][  INFO][PID:1522006] Allocated host ResallocHost, ticket_id=415264, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00667986_20251218_010558, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-18 01:16:55,247][  INFO][PID:1522006] Allocating ssh connection to builder
[2025-12-18 01:16:55,248][  INFO][PID:1522006] Checking that builder machine is OK
[2025-12-18 01:17:55,319][ ERROR][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c150 port 22: Connection timed out

[2025-12-18 01:18:05,645][ ERROR][PID:1522006] Re-try request for task on 'ResallocHost, ticket_id=415264, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00667986_20251218_010558, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-18 01:18:05,646][  INFO][PID:1522006] Releasing VM back to pool
[2025-12-18 01:18:05,656][  INFO][PID:1522006] Retry #1 (on other host)
[2025-12-18 01:18:05,657][  INFO][PID:1522006] Checking that /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/repodata/repomd.xml exists
[2025-12-18 01:18:05,657][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:05,658][  INFO][PID:1522006] VM allocation process starts
[2025-12-18 01:18:05,671][  INFO][PID:1522006] Trying to allocate VM: ResallocHost, ticket_id=415303, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-18 01:18:10,673][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:15,674][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:17,731][  INFO][PID:1522006] Allocated host ResallocHost, ticket_id=415303, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_00668134_20251218_011317, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-18 01:18:17,732][  INFO][PID:1522006] Allocating ssh connection to builder
[2025-12-18 01:18:17,733][  INFO][PID:1522006] Checking that builder machine is OK
[2025-12-18 01:18:18,766][  INFO][PID:1522006] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-12-18 01:18:19,396][  INFO][PID:1522006] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used

[2025-12-18 01:18:19,397][  INFO][PID:1522006] Filling build.info file with builder info
[2025-12-18 01:18:19,397][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:19,398][  INFO][PID:1522006] Not re-notifying FE and msg buses for the new host.
[2025-12-18 01:18:19,398][  INFO][PID:1522006] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9929001-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-12-18 01:18:20,399][  INFO][PID:1522006] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18b' (ED25519) to the list of known hosts.


[2025-12-18 01:18:20,957][  INFO][PID:1522006] Downloading the builder-live.log file, attempt 1
[2025-12-18 01:18:22,340][  INFO][PID:1522006] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-18 01:18:25,966][  INFO][PID:1522006] Periodic builder liveness probe: alive
[2025-12-18 01:18:25,967][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:30,968][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:35,969][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:40,971][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:45,972][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:50,974][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:18:55,976][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:00,977][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:06,027][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:11,029][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:16,031][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:21,035][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:26,037][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:31,039][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:36,040][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:41,042][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:46,044][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:51,047][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:19:56,150][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:01,151][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:06,152][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:11,154][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:16,155][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:21,493][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:26,494][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:31,495][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:36,496][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:41,497][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:46,499][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:51,500][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:20:56,501][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:01,502][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:06,505][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:11,506][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:16,507][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:21,508][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:26,509][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:31,512][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:36,513][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:41,514][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:46,515][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:51,519][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:21:56,521][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:01,525][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:06,527][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:11,527][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:16,529][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:21,548][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:26,557][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:31,559][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:36,561][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:41,569][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:46,576][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:51,581][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:22:56,584][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:01,586][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:06,587][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:11,592][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:16,595][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:21,599][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:26,602][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:31,604][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:36,607][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:41,609][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:46,612][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:51,616][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:23:56,618][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:01,385][ ERROR][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-18 01:24:01,623][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:06,626][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:11,390][WARNING][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-18 01:24:11,627][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:16,630][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:21,391][  INFO][PID:1522006] Downloading the builder-live.log file, attempt 2
[2025-12-18 01:24:21,398][  INFO][PID:1522006] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-18 01:24:21,633][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:26,636][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:31,639][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:36,647][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:41,648][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:46,650][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:51,658][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:24:56,665][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:01,666][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:06,668][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:11,670][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:16,673][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:21,477][ ERROR][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-18 01:25:21,680][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:26,690][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:31,502][WARNING][PID:1522006] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-18 01:25:31,692][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:36,700][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:41,505][  INFO][PID:1522006] Downloading the builder-live.log file, attempt 3
[2025-12-18 01:25:41,508][  INFO][PID:1522006] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-18 01:25:41,709][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:46,713][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:51,714][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:25:56,716][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:01,718][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:06,721][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:11,723][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:16,724][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:21,725][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:26,727][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:31,729][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:36,731][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:41,580][ ERROR][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-18 01:26:41,742][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:46,743][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:51,585][WARNING][PID:1522006] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-18 01:26:51,744][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:26:56,746][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:01,588][  INFO][PID:1522006] Downloading the builder-live.log file, attempt 4
[2025-12-18 01:27:01,592][  INFO][PID:1522006] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-18 01:27:01,748][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:06,749][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:11,751][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:16,752][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:21,756][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:26,759][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:31,761][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:36,762][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:41,763][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:46,765][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:51,767][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:27:56,771][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:28:01,665][ ERROR][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-18 01:28:01,772][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:28:06,773][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:28:11,669][ ERROR][PID:1522006] Giving up for unstable SSH, failures: 1766021051.3901894, 1766021131.5027757, 1766021211.5857825, 1766021291.6694136
[2025-12-18 01:28:11,671][ ERROR][PID:1522006] Re-try request for task on 'ResallocHost, ticket_id=415303, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_00668134_20251218_011317, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-18 01:28:11,674][  INFO][PID:1522006] Releasing VM back to pool
[2025-12-18 01:28:11,719][  INFO][PID:1522006] Retry #2 (on other host)
[2025-12-18 01:28:11,721][  INFO][PID:1522006] Checking that /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/repodata/repomd.xml exists
[2025-12-18 01:28:11,723][  INFO][PID:1522006] Checking for cancel request
[2025-12-18 01:28:11,727][  INFO][PID:1522006] VM allocation process starts
[2025-12-18 01:28:11,776][  INFO][PID:1522006] Trying to allocate VM: ResallocHost, ticket_id=415668, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-18 01:28:14,805][  INFO][PID:1522006] Allocated host ResallocHost, ticket_id=415668, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00668186_20251218_011716, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-18 01:28:14,806][  INFO][PID:1522006] Allocating ssh connection to builder
[2025-12-18 01:28:14,807][  INFO][PID:1522006] Checking that builder machine is OK
[2025-12-18 01:29:14,862][ ERROR][PID:1522006] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18c port 22: Connection timed out

[2025-12-18 01:29:24,878][ ERROR][PID:1522006] Re-try request for task on 'ResallocHost, ticket_id=415668, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_00668186_20251218_011716, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-18 01:29:24,879][  INFO][PID:1522006] Releasing VM back to pool
[2025-12-18 01:29:24,895][ ERROR][PID:1522006] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c18c', '2620:52:6:1161:dead:beef:cafe:c150', '2620:52:6:1161:dead:beef:cafe:c18b'}
[2025-12-18 01:29:24,897][WARNING][PID:1522006] Switching not-finished job state to 'failed'
[2025-12-18 01:29:24,898][  INFO][PID:1522006] Worker failed build, took 665.4988152980804
[2025-12-18 01:29:24,899][  INFO][PID:1522006] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "rpmsoftwaremanagement",
            "project_name": "dnf-nightly",
            "project_dirname": "dnf-nightly",
            "submitter": "rpmsoftwaremanagement",
            "ended_on": 1766021364.8972447,
            "started_on": 1766020699.3984294,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-41-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9929001-fedora-41-ppc64le",
            "build_id": 9929001,
            "package_name": "microdnf",
            "package_version": "3.10.1-1.20251218011418336114.master.3.g0c194e8",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rpmsoftwaremanagement/dnf-nightly/microdnf",
            "git_hash": "a0fea2ad41bb30a7706b93158e9b2710ea3f9d85",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "rpmsoftwaremanagement/dnf-nightly--rpmsoftwaremanagement",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly",
            "result_dir": "09929001-microdnf",
            "built_packages": "",
            "id": 9929001,
            "mockchain_macros": {
                "copr_username": "rpmsoftwaremanagement",
                "copr_projectname": "dnf-nightly",
                "vendor": "Fedora Project COPR (rpmsoftwaremanagement/dnf-nightly)"
            }
        }
    ]
}
[2025-12-18 01:29:25,199][  INFO][PID:1522006] Sending fedora-messaging bus message in build.end
[2025-12-18 01:29:25,888][  INFO][PID:1522006] Compressing /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09929001-microdnf/builder-live.log by gzip
[2025-12-18 01:29:25,889][  INFO][PID:1522006] Running command 'gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09929001-microdnf/builder-live.log' as PID 1655461
[2025-12-18 01:29:25,893][  INFO][PID:1522006] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09929001-microdnf/builder-live.log)
[2025-12-18 01:29:25,894][  INFO][PID:1522006] Compressing /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09929001-microdnf/backend.log by gzip
[2025-12-18 01:29:25,895][  INFO][PID:1522006] Running command 'gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09929001-microdnf/backend.log' as PID 1655462