[2025-12-16 03:47:04,990][  INFO][PID:1164028] Marking build as starting
[2025-12-16 03:47:05,197][  INFO][PID:1164028] Checking that /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-16 03:47:05,234][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:47:05,235][  INFO][PID:1164028] VM allocation process starts
[2025-12-16 03:47:05,303][  INFO][PID:1164028] Trying to allocate VM: ResallocHost, ticket_id=363101, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 03:47:08,364][  INFO][PID:1164028] Allocated host ResallocHost, ticket_id=363101, hostname=2620:52:6:1161:dead:beef:cafe:c156, name=vmhost_p09_03_prod_00588806_20251216_032727, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 03:47:08,365][  INFO][PID:1164028] Allocating ssh connection to builder
[2025-12-16 03:47:08,365][  INFO][PID:1164028] Checking that builder machine is OK
[2025-12-16 03:48:08,453][ ERROR][PID:1164028] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c156 port 22: Connection timed out

[2025-12-16 03:48:18,457][ ERROR][PID:1164028] Re-try request for task on 'ResallocHost, ticket_id=363101, hostname=2620:52:6:1161:dead:beef:cafe:c156, name=vmhost_p09_03_prod_00588806_20251216_032727, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-16 03:48:18,458][  INFO][PID:1164028] Releasing VM back to pool
[2025-12-16 03:48:18,479][  INFO][PID:1164028] Retry #1 (on other host)
[2025-12-16 03:48:18,480][  INFO][PID:1164028] Checking that /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-16 03:48:18,481][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:48:18,484][  INFO][PID:1164028] VM allocation process starts
[2025-12-16 03:48:18,511][  INFO][PID:1164028] Trying to allocate VM: ResallocHost, ticket_id=363133, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 03:48:23,513][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:48:24,578][  INFO][PID:1164028] Allocated host ResallocHost, ticket_id=363133, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00588898_20251216_033025, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 03:48:24,578][  INFO][PID:1164028] Allocating ssh connection to builder
[2025-12-16 03:48:24,581][  INFO][PID:1164028] Checking that builder machine is OK
[2025-12-16 03:48:25,512][  INFO][PID:1164028] Running remote command: copr-builder-ready fedora-43-ppc64le
[2025-12-16 03:49:25,614][ ERROR][PID:1164028] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c148 port 22: Connection timed out

[2025-12-16 03:49:35,616][ ERROR][PID:1164028] Re-try request for task on 'ResallocHost, ticket_id=363133, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00588898_20251216_033025, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-16 03:49:35,625][  INFO][PID:1164028] Releasing VM back to pool
[2025-12-16 03:49:35,661][  INFO][PID:1164028] Retry #2 (on other host)
[2025-12-16 03:49:35,665][  INFO][PID:1164028] Checking that /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/repodata/repomd.xml exists
[2025-12-16 03:49:35,671][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:49:35,676][  INFO][PID:1164028] VM allocation process starts
[2025-12-16 03:49:35,702][  INFO][PID:1164028] Trying to allocate VM: ResallocHost, ticket_id=363164, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 03:49:40,704][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:49:41,749][  INFO][PID:1164028] Allocated host ResallocHost, ticket_id=363164, hostname=2620:52:6:1161:dead:beef:cafe:c196, name=vmhost_p09_04_prod_00588872_20251216_032908, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-16 03:49:41,751][  INFO][PID:1164028] Allocating ssh connection to builder
[2025-12-16 03:49:41,752][  INFO][PID:1164028] Checking that builder machine is OK
[2025-12-16 03:49:42,763][  INFO][PID:1164028] Running remote command: copr-builder-ready fedora-43-ppc64le
[2025-12-16 03:49:43,318][  INFO][PID:1164028] Red Hat subscription not needed for fedora-43-ppc64le
Builder is ready to be used

[2025-12-16 03:49:43,320][  INFO][PID:1164028] Filling build.info file with builder info
[2025-12-16 03:49:43,321][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:49:43,323][  INFO][PID:1164028] Not re-notifying FE and msg buses for the new host.
[2025-12-16 03:49:43,324][  INFO][PID:1164028] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9913736-fedora-43-ppc64le --chroot fedora-43-ppc64le --detached
[2025-12-16 03:49:44,145][  INFO][PID:1164028] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c196' (ED25519) to the list of known hosts.


[2025-12-16 03:49:44,146][  INFO][PID:1164028] Downloading the builder-live.log file, attempt 1
[2025-12-16 03:49:44,148][  INFO][PID:1164028] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c196 copr-rpmbuild-log
[2025-12-16 03:49:49,155][  INFO][PID:1164028] Periodic builder liveness probe: alive
[2025-12-16 03:49:49,156][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:49:54,157][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:49:59,160][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:04,162][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:09,165][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:14,170][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:19,172][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:24,173][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:29,180][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:34,184][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:39,185][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:44,187][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:49,188][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:54,189][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:50:59,190][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:04,191][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:09,192][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:14,193][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:19,194][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:24,195][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:29,196][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:34,197][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:39,198][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:44,200][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:49,201][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:54,202][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:51:59,204][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:04,213][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:09,214][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:14,215][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:19,216][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:24,217][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:29,220][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:34,221][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:39,222][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:44,226][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:49,227][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:54,228][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:52:59,229][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:04,230][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:09,232][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:14,233][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:19,234][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:24,235][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:29,239][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:34,241][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:39,243][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:44,244][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:49,254][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:54,256][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:53:59,257][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:04,258][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:09,259][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:14,261][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:19,263][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:24,264][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:29,265][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:34,267][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:39,268][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:44,269][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:49,270][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:54,274][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:54:59,278][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:04,288][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:09,290][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:14,291][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:19,292][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:24,293][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:29,296][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:34,297][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:39,299][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:44,305][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:49,309][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:54,311][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:55:59,314][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:04,317][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:09,319][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:14,324][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:19,330][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:24,331][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:29,334][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:34,336][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:39,337][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:44,341][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:49,342][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:54,345][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:56:59,347][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:04,348][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:09,351][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:14,353][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:19,354][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:24,356][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:29,358][  INFO][PID:1164028] Checking for cancel request
[2025-12-16 03:57:29,915][  INFO][PID:1164028] Downloading results from builder
[2025-12-16 03:57:29,915][  INFO][PID:1164028] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5 started
[2025-12-16 03:57:29,922][  INFO][PID:1164028] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/ &> /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/build-09913736.rsync.log
[2025-12-16 03:57:30,618][  INFO][PID:1164028] rsyncing finished.
[2025-12-16 03:57:30,618][  INFO][PID:1164028] Releasing VM back to pool
[2025-12-16 03:57:30,641][  INFO][PID:1164028] Searching for 'success' file in resultdir
[2025-12-16 03:57:30,645][ ERROR][PID:1164028] Build failed: Backend process error: No success file => build failure
[2025-12-16 03:57:30,646][  INFO][PID:1164028] Finished build: id=9913736 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly chroot=fedora-43-ppc64le 
[2025-12-16 03:57:30,650][ ERROR][PID:1164028] Backend process error: No success file => build failure
[2025-12-16 03:57:30,652][  INFO][PID:1164028] Worker failed build, took 467.3282587528229
[2025-12-16 03:57:30,653][  INFO][PID:1164028] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "rpmsoftwaremanagement",
            "project_name": "dnf-nightly",
            "project_dirname": "dnf-nightly",
            "submitter": "rpmsoftwaremanagement",
            "ended_on": 1765857450.651972,
            "started_on": 1765856983.3237133,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-43-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9913736-fedora-43-ppc64le",
            "build_id": 9913736,
            "package_name": "dnf5",
            "package_version": "5.3.0.0-1.20251216034558521395.main.60.g0c30f111",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rpmsoftwaremanagement/dnf-nightly/dnf5",
            "git_hash": "549745462c95d9c26845b3c6aa36f753630c0435",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "rpmsoftwaremanagement/dnf-nightly--rpmsoftwaremanagement",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly",
            "result_dir": "09913736-dnf5",
            "built_packages": "",
            "id": 9913736,
            "mockchain_macros": {
                "copr_username": "rpmsoftwaremanagement",
                "copr_projectname": "dnf-nightly",
                "vendor": "Fedora Project COPR (rpmsoftwaremanagement/dnf-nightly)"
            }
        }
    ]
}
[2025-12-16 03:57:30,743][  INFO][PID:1164028] Sending fedora-messaging bus message in build.end
[2025-12-16 03:57:31,499][  INFO][PID:1164028] Compressing /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/builder-live.log by gzip
[2025-12-16 03:57:31,500][  INFO][PID:1164028] Running command 'gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/builder-live.log' as PID 1251164
[2025-12-16 03:57:31,566][  INFO][PID:1164028] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/builder-live.log)
[2025-12-16 03:57:31,567][  INFO][PID:1164028] Compressing /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/backend.log by gzip
[2025-12-16 03:57:31,569][  INFO][PID:1164028] Running command 'gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-43-ppc64le/09913736-dnf5/backend.log' as PID 1251186