[2025-12-11 13:36:05,763][  INFO][PID:2573576] Marking build as starting
[2025-12-11 13:36:05,811][  INFO][PID:2573576] Checking that /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/repodata/repomd.xml exists
[2025-12-11 13:36:05,813][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:36:05,816][  INFO][PID:2573576] VM allocation process starts
[2025-12-11 13:36:05,834][  INFO][PID:2573576] Trying to allocate VM: ResallocHost, ticket_id=283795, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-11 13:36:10,836][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:36:11,860][  INFO][PID:2573576] Allocated host ResallocHost, ticket_id=283795, hostname=2620:52:6:1161:dead:beef:cafe:c188, name=vmhost_p09_04_prod_00346610_20251211_132733, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-11 13:36:11,861][  INFO][PID:2573576] Allocating ssh connection to builder
[2025-12-11 13:36:11,861][  INFO][PID:2573576] Checking that builder machine is OK
[2025-12-11 13:37:11,876][ ERROR][PID:2573576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c188 port 22: Connection timed out

[2025-12-11 13:37:21,877][ ERROR][PID:2573576] Re-try request for task on 'ResallocHost, ticket_id=283795, hostname=2620:52:6:1161:dead:beef:cafe:c188, name=vmhost_p09_04_prod_00346610_20251211_132733, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-11 13:37:21,878][  INFO][PID:2573576] Releasing VM back to pool
[2025-12-11 13:37:21,890][  INFO][PID:2573576] Retry #1 (on other host)
[2025-12-11 13:37:21,890][  INFO][PID:2573576] Checking that /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/repodata/repomd.xml exists
[2025-12-11 13:37:21,891][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:37:21,891][  INFO][PID:2573576] VM allocation process starts
[2025-12-11 13:37:21,902][  INFO][PID:2573576] Trying to allocate VM: ResallocHost, ticket_id=283802, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-11 13:37:24,916][  INFO][PID:2573576] Allocated host ResallocHost, ticket_id=283802, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_00346737_20251211_133227, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-11 13:37:24,917][  INFO][PID:2573576] Allocating ssh connection to builder
[2025-12-11 13:37:24,917][  INFO][PID:2573576] Checking that builder machine is OK
[2025-12-11 13:38:24,983][ ERROR][PID:2573576] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c195 port 22: Connection timed out

[2025-12-11 13:38:34,984][ ERROR][PID:2573576] Re-try request for task on 'ResallocHost, ticket_id=283802, hostname=2620:52:6:1161:dead:beef:cafe:c195, name=vmhost_p09_04_prod_00346737_20251211_133227, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2025-12-11 13:38:34,985][  INFO][PID:2573576] Releasing VM back to pool
[2025-12-11 13:38:34,996][  INFO][PID:2573576] Retry #2 (on other host)
[2025-12-11 13:38:34,997][  INFO][PID:2573576] Checking that /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/repodata/repomd.xml exists
[2025-12-11 13:38:34,997][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:38:34,998][  INFO][PID:2573576] VM allocation process starts
[2025-12-11 13:38:35,009][  INFO][PID:2573576] Trying to allocate VM: ResallocHost, ticket_id=283810, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-11 13:38:40,010][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:38:41,025][  INFO][PID:2573576] Allocated host ResallocHost, ticket_id=283810, hostname=2620:52:6:1161:dead:beef:cafe:c10f, name=vmhost_p09_02_prod_00346686_20251211_133104, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2025-12-11 13:38:41,025][  INFO][PID:2573576] Allocating ssh connection to builder
[2025-12-11 13:38:41,026][  INFO][PID:2573576] Checking that builder machine is OK
[2025-12-11 13:38:42,020][  INFO][PID:2573576] Running remote command: copr-builder-ready centos-stream-10-ppc64le
[2025-12-11 13:38:42,560][  INFO][PID:2573576] Red Hat subscription not needed for centos-stream-10-ppc64le
Builder is ready to be used

[2025-12-11 13:38:42,561][  INFO][PID:2573576] Filling build.info file with builder info
[2025-12-11 13:38:42,562][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:38:42,563][  INFO][PID:2573576] Not re-notifying FE and msg buses for the new host.
[2025-12-11 13:38:42,564][  INFO][PID:2573576] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9899285-centos-stream-10-ppc64le --chroot centos-stream-10-ppc64le --detached
[2025-12-11 13:38:43,614][  INFO][PID:2573576] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10f' (ED25519) to the list of known hosts.


[2025-12-11 13:38:43,615][  INFO][PID:2573576] Downloading the builder-live.log file, attempt 1
[2025-12-11 13:38:43,616][  INFO][PID:2573576] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10f copr-rpmbuild-log
[2025-12-11 13:38:48,621][  INFO][PID:2573576] Periodic builder liveness probe: alive
[2025-12-11 13:38:48,622][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:38:53,623][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:38:58,624][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:03,625][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:08,626][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:13,627][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:18,628][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:23,629][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:28,630][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:33,631][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:38,632][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:43,633][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:48,634][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:53,635][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:39:58,636][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:03,637][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:08,638][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:13,639][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:18,640][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:23,641][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:28,642][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:33,643][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:38,644][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:43,645][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:48,646][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:53,646][  INFO][PID:2573576] Checking for cancel request
[2025-12-11 13:40:57,295][  INFO][PID:2573576] Downloading results from builder
[2025-12-11 13:40:57,295][  INFO][PID:2573576] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c10f]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc started
[2025-12-11 13:40:57,296][  INFO][PID:2573576] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c10f]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/ &> /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/build-09899285.rsync.log
[2025-12-11 13:41:57,370][  INFO][PID:2573576] rsyncing finished.
[2025-12-11 13:41:57,370][ ERROR][PID:2573576] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-11 13:41:57,371][ ERROR][PID:2573576] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-11 13:42:07,371][  INFO][PID:2573576] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c10f]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc started
[2025-12-11 13:42:07,372][  INFO][PID:2573576] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c10f]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/ &> /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/build-09899285.rsync.log
[2025-12-11 13:43:07,448][  INFO][PID:2573576] rsyncing finished.
[2025-12-11 13:43:07,448][ ERROR][PID:2573576] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-11 13:43:07,449][ ERROR][PID:2573576] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-11 13:43:17,449][  INFO][PID:2573576] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c10f]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc started
[2025-12-11 13:43:17,450][  INFO][PID:2573576] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c10f]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/ &> /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/build-09899285.rsync.log
[2025-12-11 13:44:17,527][  INFO][PID:2573576] rsyncing finished.
[2025-12-11 13:44:17,528][ ERROR][PID:2573576] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-11 13:44:17,529][ ERROR][PID:2573576] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2025-12-11 13:44:27,529][ ERROR][PID:2573576] Re-try request for task on 'ResallocHost, ticket_id=283810, hostname=2620:52:6:1161:dead:beef:cafe:c10f, name=vmhost_p09_02_prod_00346686_20251211_133104, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts
[2025-12-11 13:44:27,530][  INFO][PID:2573576] Releasing VM back to pool
[2025-12-11 13:44:27,541][ ERROR][PID:2573576] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c10f', '2620:52:6:1161:dead:beef:cafe:c188', '2620:52:6:1161:dead:beef:cafe:c195'}
[2025-12-11 13:44:27,542][WARNING][PID:2573576] Switching not-finished job state to 'failed'
[2025-12-11 13:44:27,543][  INFO][PID:2573576] Worker failed build, took 344.9793770313263
[2025-12-11 13:44:27,543][  INFO][PID:2573576] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "osloup",
            "project_name": "docbook5_style_xsl_license_and_jar_build",
            "project_dirname": "docbook5_style_xsl_license_and_jar_build",
            "submitter": "https://src.fedoraproject.org/user/jakub",
            "ended_on": 1765460667.5424848,
            "started_on": 1765460322.5631077,
            "submitted_on": null,
            "status": 0,
            "chroot": "centos-stream-10-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9899285-centos-stream-10-ppc64le",
            "build_id": 9899285,
            "package_name": "gcc",
            "package_version": "15.2.1-5",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/docbook5_style_xsl_license_and_jar_build/gcc",
            "git_hash": "b4ce49277f8b538f2550e01d3179102adad1765a",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "osloup/docbook5_style_xsl_license_and_jar_build--https://src.fedoraproject.org/user/jakub",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/docbook5_style_xsl_license_and_jar_build",
            "result_dir": "09899285-gcc",
            "built_packages": "",
            "id": 9899285,
            "mockchain_macros": {
                "copr_username": "osloup",
                "copr_projectname": "docbook5_style_xsl_license_and_jar_build",
                "vendor": "Fedora Project COPR (osloup/docbook5_style_xsl_license_and_jar_build)"
            }
        }
    ]
}
[2025-12-11 13:44:27,590][  INFO][PID:2573576] Sending fedora-messaging bus message in build.end
[2025-12-11 13:44:28,201][  INFO][PID:2573576] Compressing /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/builder-live.log by gzip
[2025-12-11 13:44:28,203][  INFO][PID:2573576] Running command 'gzip /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/builder-live.log' as PID 2610047
[2025-12-11 13:44:28,206][  INFO][PID:2573576] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/builder-live.log)
[2025-12-11 13:44:28,207][  INFO][PID:2573576] Compressing /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/backend.log by gzip
[2025-12-11 13:44:28,208][  INFO][PID:2573576] Running command 'gzip /var/lib/copr/public_html/results/osloup/docbook5_style_xsl_license_and_jar_build/centos-stream-10-ppc64le/09899285-gcc/backend.log' as PID 2610048