[2025-12-25 00:05:50,907][  INFO][PID:3695160] Marking build as starting
[2025-12-25 00:05:51,423][  INFO][PID:3695160] Checking that /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-25 00:05:51,453][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:05:51,454][  INFO][PID:3695160] VM allocation process starts
[2025-12-25 00:05:51,464][  INFO][PID:3695160] Trying to allocate VM: ResallocHost, ticket_id=499651, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-25 00:05:56,466][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:05:57,486][  INFO][PID:3695160] Allocated host ResallocHost, ticket_id=499651, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_00756627_20251224_235947, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-25 00:05:57,487][  INFO][PID:3695160] Allocating ssh connection to builder
[2025-12-25 00:05:57,487][  INFO][PID:3695160] Checking that builder machine is OK
[2025-12-25 00:06:57,557][ ERROR][PID:3695160] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18b port 22: Connection timed out

[2025-12-25 00:07:07,587][ ERROR][PID:3695160] Re-try request for task on 'ResallocHost, ticket_id=499651, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_00756627_20251224_235947, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-25 00:07:07,588][  INFO][PID:3695160] Releasing VM back to pool
[2025-12-25 00:07:07,597][  INFO][PID:3695160] Retry #1 (on other host)
[2025-12-25 00:07:07,597][  INFO][PID:3695160] Checking that /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-25 00:07:07,598][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:07,598][  INFO][PID:3695160] VM allocation process starts
[2025-12-25 00:07:07,630][  INFO][PID:3695160] Trying to allocate VM: ResallocHost, ticket_id=499688, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-25 00:07:10,655][  INFO][PID:3695160] Allocated host ResallocHost, ticket_id=499688, hostname=2620:52:6:1161:dead:beef:cafe:c196, name=vmhost_p09_04_prod_00756643_20251225_000155, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-25 00:07:10,656][  INFO][PID:3695160] Allocating ssh connection to builder
[2025-12-25 00:07:10,656][  INFO][PID:3695160] Checking that builder machine is OK
[2025-12-25 00:07:11,551][  INFO][PID:3695160] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-12-25 00:07:12,030][  INFO][PID:3695160] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-12-25 00:07:12,031][  INFO][PID:3695160] Filling build.info file with builder info
[2025-12-25 00:07:12,032][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:12,032][  INFO][PID:3695160] Not re-notifying FE and msg buses for the new host.
[2025-12-25 00:07:12,033][  INFO][PID:3695160] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9948598-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-12-25 00:07:12,954][  INFO][PID:3695160] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c196' (ED25519) to the list of known hosts.


[2025-12-25 00:07:12,968][  INFO][PID:3695160] Downloading the builder-live.log file, attempt 1
[2025-12-25 00:07:12,969][  INFO][PID:3695160] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c196 copr-rpmbuild-log
[2025-12-25 00:07:17,973][  INFO][PID:3695160] Periodic builder liveness probe: alive
[2025-12-25 00:07:17,973][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:22,974][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:27,975][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:32,976][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:37,977][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:42,978][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:47,979][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:52,983][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:07:57,985][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:02,990][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:07,992][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:12,994][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:17,995][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:22,997][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:27,999][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:33,003][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:38,009][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:43,011][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:48,013][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:53,014][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:08:58,015][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:03,017][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:08,019][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:13,021][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:18,024][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:23,027][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:28,031][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:33,033][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:38,041][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:43,042][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:48,045][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:53,052][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:09:58,054][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:03,059][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:08,062][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:13,065][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:18,067][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:23,069][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:28,073][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:33,074][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:38,076][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:43,077][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:48,079][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:53,081][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:10:58,083][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:03,084][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:08,086][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:13,087][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:18,091][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:23,094][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:28,095][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:33,100][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:38,101][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:43,104][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:48,107][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:53,109][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:11:58,120][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:03,126][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:08,128][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:13,136][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:18,137][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:23,143][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:28,146][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:33,164][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:38,170][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:43,176][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:48,178][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:53,187][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:12:58,191][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:13:03,192][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:13:08,194][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:13:13,195][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:13:18,197][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:13:20,779][  INFO][PID:3695160] Downloading results from builder
[2025-12-25 00:13:20,780][  INFO][PID:3695160] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect started
[2025-12-25 00:13:20,781][  INFO][PID:3695160] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/ &> /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/build-09948598.rsync.log
[2025-12-25 00:14:20,890][  INFO][PID:3695160] rsyncing finished.
[2025-12-25 00:14:20,891][ ERROR][PID:3695160] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-25 00:14:20,892][ ERROR][PID:3695160] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-25 00:14:30,892][  INFO][PID:3695160] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect started
[2025-12-25 00:14:30,893][  INFO][PID:3695160] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/ &> /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/build-09948598.rsync.log
[2025-12-25 00:15:30,974][  INFO][PID:3695160] rsyncing finished.
[2025-12-25 00:15:30,974][ ERROR][PID:3695160] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-25 00:15:30,975][ ERROR][PID:3695160] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-25 00:15:40,976][  INFO][PID:3695160] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect started
[2025-12-25 00:15:40,977][  INFO][PID:3695160] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c196]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/ &> /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/build-09948598.rsync.log
[2025-12-25 00:16:41,070][  INFO][PID:3695160] rsyncing finished.
[2025-12-25 00:16:41,070][ ERROR][PID:3695160] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-25 00:16:41,071][ ERROR][PID:3695160] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-25 00:16:51,071][ ERROR][PID:3695160] Re-try request for task on 'ResallocHost, ticket_id=499688, hostname=2620:52:6:1161:dead:beef:cafe:c196, name=vmhost_p09_04_prod_00756643_20251225_000155, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 3 SSH attempts
[2025-12-25 00:16:51,072][  INFO][PID:3695160] Releasing VM back to pool
[2025-12-25 00:16:51,085][  INFO][PID:3695160] Retry #2 (on other host)
[2025-12-25 00:16:51,085][  INFO][PID:3695160] Checking that /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-25 00:16:51,086][  INFO][PID:3695160] Checking for cancel request
[2025-12-25 00:16:51,087][  INFO][PID:3695160] VM allocation process starts
[2025-12-25 00:16:51,098][  INFO][PID:3695160] Trying to allocate VM: ResallocHost, ticket_id=499797, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-25 00:16:54,111][  INFO][PID:3695160] Allocated host ResallocHost, ticket_id=499797, hostname=2620:52:6:1161:dead:beef:cafe:c153, name=vmhost_p09_03_prod_00756667_20251225_000423, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-25 00:16:54,111][  INFO][PID:3695160] Allocating ssh connection to builder
[2025-12-25 00:16:54,112][  INFO][PID:3695160] Checking that builder machine is OK
[2025-12-25 00:17:54,180][ ERROR][PID:3695160] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c153 port 22: Connection timed out

[2025-12-25 00:18:04,181][ ERROR][PID:3695160] Re-try request for task on 'ResallocHost, ticket_id=499797, hostname=2620:52:6:1161:dead:beef:cafe:c153, name=vmhost_p09_03_prod_00756667_20251225_000423, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-25 00:18:04,181][  INFO][PID:3695160] Releasing VM back to pool
[2025-12-25 00:18:04,193][ ERROR][PID:3695160] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c196', '2620:52:6:1161:dead:beef:cafe:c18b', '2620:52:6:1161:dead:beef:cafe:c153'}
[2025-12-25 00:18:04,195][WARNING][PID:3695160] Switching not-finished job state to 'failed'
[2025-12-25 00:18:04,198][  INFO][PID:3695160] Worker failed build, took 652.1621391773224
[2025-12-25 00:18:04,199][  INFO][PID:3695160] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "dwmw2",
            "project_name": "openconnect",
            "project_dirname": "openconnect",
            "submitter": null,
            "ended_on": 1766621884.1950126,
            "started_on": 1766621232.0328734,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9948598-fedora-rawhide-ppc64le",
            "build_id": 9948598,
            "package_name": "NetworkManager-openconnect",
            "package_version": "1.2.10.git.80.89bc23d-0",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dwmw2/openconnect/NetworkManager-openconnect",
            "git_hash": "80048c118a5fe532f496d0e16a55fbceee4f0656",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dwmw2/openconnect--29a97d02-901b-49b4-87d1-07680e8c7f66",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dwmw2/openconnect/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dwmw2/openconnect",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dwmw2/openconnect",
            "result_dir": "09948598-NetworkManager-openconnect",
            "built_packages": "",
            "id": 9948598,
            "mockchain_macros": {
                "copr_username": "dwmw2",
                "copr_projectname": "openconnect",
                "vendor": "Fedora Project COPR (dwmw2/openconnect)"
            }
        }
    ]
}
[2025-12-25 00:18:04,276][  INFO][PID:3695160] Sending fedora-messaging bus message in build.end
[2025-12-25 00:18:04,934][  INFO][PID:3695160] Compressing /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/builder-live.log by gzip
[2025-12-25 00:18:04,936][  INFO][PID:3695160] Running command 'gzip /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/builder-live.log' as PID 3759813
[2025-12-25 00:18:04,956][  INFO][PID:3695160] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/builder-live.log)
[2025-12-25 00:18:04,957][  INFO][PID:3695160] Compressing /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/backend.log by gzip
[2025-12-25 00:18:04,960][  INFO][PID:3695160] Running command 'gzip /var/lib/copr/public_html/results/dwmw2/openconnect/fedora-rawhide-ppc64le/09948598-NetworkManager-openconnect/backend.log' as PID 3759815