[2025-12-10 22:09:43,686][ INFO][PID:1529662] Marking build as starting [2025-12-10 22:09:43,732][ INFO][PID:1529662] Checking that /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/repodata/repomd.xml exists [2025-12-10 22:09:43,733][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:09:43,734][ INFO][PID:1529662] VM allocation process starts [2025-12-10 22:09:43,749][ INFO][PID:1529662] Trying to allocate VM: ResallocHost, ticket_id=269362, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 22:09:48,751][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:09:53,752][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:09:55,803][ INFO][PID:1529662] Allocated host ResallocHost, ticket_id=269362, hostname=2620:52:6:1161:dead:beef:cafe:c193, name=vmhost_p09_04_prod_00331829_20251210_211701, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2025-12-10 22:09:55,803][ INFO][PID:1529662] Allocating ssh connection to builder [2025-12-10 22:09:55,804][ INFO][PID:1529662] Checking that builder machine is OK [2025-12-10 22:09:56,839][ INFO][PID:1529662] Running remote command: copr-builder-ready centos-stream-9-ppc64le [2025-12-10 22:09:57,326][ INFO][PID:1529662] Red Hat subscription not needed for centos-stream-9-ppc64le Builder is ready to be used [2025-12-10 22:09:57,326][ INFO][PID:1529662] Filling build.info file with builder info [2025-12-10 22:09:57,328][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:09:57,329][ INFO][PID:1529662] Sending build state back to frontend: { "builds": [ { "timeout": 90000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@fedora-llvm-team", "project_name": "clang-fedora-centos-testing", "project_dirname": "clang-fedora-centos-testing", "submitter": "tstellar", "ended_on": null, "started_on": 1765404597.3294835, "submitted_on": null, "status": 3, "chroot": "centos-stream-9-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9896379-centos-stream-9-ppc64le", "build_id": 9896379, "package_name": "thunderbird", "package_version": "140.5.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/clang-fedora-centos-testing/thunderbird", "git_hash": "093b8ea89a67f712ba59362aaf49a2db00452412", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@fedora-llvm-team/clang-fedora-centos-testing--tstellar", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": 1000 }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-compat-packages/centos-stream-9-ppc64le/", "id": "copr_fedora_llvm_team_llvm_compat_packages", "name": "Additional repo copr_fedora_llvm_team_llvm_compat_packages" }, { "baseurl": "https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/ppc64le/", "id": "https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le", "name": "Additional repo https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le" }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20251209/centos-stream-9-ppc64le/", "id": "copr_fedora_llvm_team_llvm_snapshots_big_merge_20251209", "module_hotfixes": true, "name": "Additional repo copr_fedora_llvm_team_llvm_snapshots_big_merge_20251209" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": 1000, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing", "result_dir": "09896379-thunderbird", "built_packages": "", "id": 9896379, "mockchain_macros": { "copr_username": "@fedora-llvm-team", "copr_projectname": "clang-fedora-centos-testing", "vendor": "Fedora Project COPR (@fedora-llvm-team/clang-fedora-centos-testing)" } } ] } [2025-12-10 22:09:57,365][ INFO][PID:1529662] Sending fedora-messaging bus message in build.start [2025-12-10 22:09:57,983][ INFO][PID:1529662] Sending fedora-messaging bus message in chroot.start [2025-12-10 22:09:58,011][ INFO][PID:1529662] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9896379-centos-stream-9-ppc64le --chroot centos-stream-9-ppc64le --detached [2025-12-10 22:09:58,841][ INFO][PID:1529662] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c193' (ED25519) to the list of known hosts. [2025-12-10 22:09:58,842][ INFO][PID:1529662] Downloading the builder-live.log file, attempt 1 [2025-12-10 22:09:58,845][ INFO][PID:1529662] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c193 copr-rpmbuild-log [2025-12-10 22:10:03,861][ INFO][PID:1529662] Periodic builder liveness probe: alive [2025-12-10 22:10:03,862][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:08,863][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:13,864][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:18,866][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:23,867][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:28,868][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:33,869][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:38,870][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:43,871][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:48,873][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:53,874][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:10:58,875][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:03,876][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:08,877][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:13,878][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:18,879][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:23,880][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:28,882][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:33,883][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:38,884][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:43,885][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:48,890][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:53,891][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:11:58,892][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:03,893][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:08,895][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:13,896][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:18,897][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:23,898][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:28,899][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:33,900][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:38,901][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:43,905][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:48,907][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:53,908][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:12:58,909][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:03,910][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:08,911][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:13,912][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:18,913][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:23,914][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:28,915][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:33,916][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:38,917][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:43,918][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:48,919][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:53,921][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:13:58,922][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:03,923][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:08,924][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:13,925][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:18,926][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:23,927][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:28,928][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:33,929][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:38,930][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:43,931][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:48,933][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:53,934][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:14:58,935][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:03,936][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:08,937][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:13,938][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:18,939][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:23,940][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:28,941][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:33,942][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:38,943][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:43,950][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:48,951][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:53,952][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:15:58,953][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:03,954][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:08,955][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:13,956][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:18,963][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:23,964][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:28,971][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:33,978][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:38,979][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:43,981][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:48,982][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:53,989][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:16:58,991][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:03,992][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:08,993][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:13,994][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:18,996][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:23,997][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:28,998][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:33,999][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:39,000][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:44,002][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:49,003][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:54,004][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:17:59,005][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:04,006][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:09,012][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:14,013][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:19,014][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:24,015][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:29,017][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:34,018][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:39,019][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:44,020][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:49,021][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:54,022][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:18:59,024][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:04,024][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:09,026][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:14,027][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:19,028][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:24,029][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:29,030][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:34,031][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:39,032][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:44,033][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:49,034][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:54,035][ INFO][PID:1529662] Checking for cancel request [2025-12-10 22:19:57,548][ INFO][PID:1529662] Downloading results from builder [2025-12-10 22:19:57,549][ INFO][PID:1529662] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c193]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird started [2025-12-10 22:19:57,550][ INFO][PID:1529662] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c193]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/build-09896379.rsync.log [2025-12-10 22:20:05,623][ INFO][PID:1529662] rsyncing finished. [2025-12-10 22:20:05,624][ INFO][PID:1529662] Releasing VM back to pool [2025-12-10 22:20:05,634][ INFO][PID:1529662] Searching for 'success' file in resultdir [2025-12-10 22:20:05,635][ ERROR][PID:1529662] Build failed: Backend process error: No success file => build failure [2025-12-10 22:20:05,635][ INFO][PID:1529662] Finished build: id=9896379 failed=True timeout=90000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing chroot=centos-stream-9-ppc64le [2025-12-10 22:20:05,637][ ERROR][PID:1529662] Backend process error: No success file => build failure [2025-12-10 22:20:05,637][ INFO][PID:1529662] Worker failed build, took 608.3084285259247 [2025-12-10 22:20:05,638][ INFO][PID:1529662] Sending build state back to frontend: { "builds": [ { "timeout": 90000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@fedora-llvm-team", "project_name": "clang-fedora-centos-testing", "project_dirname": "clang-fedora-centos-testing", "submitter": "tstellar", "ended_on": 1765405205.637912, "started_on": 1765404597.3294835, "submitted_on": null, "status": 0, "chroot": "centos-stream-9-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9896379-centos-stream-9-ppc64le", "build_id": 9896379, "package_name": "thunderbird", "package_version": "140.5.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/clang-fedora-centos-testing/thunderbird", "git_hash": "093b8ea89a67f712ba59362aaf49a2db00452412", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@fedora-llvm-team/clang-fedora-centos-testing--tstellar", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": 1000 }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-compat-packages/centos-stream-9-ppc64le/", "id": "copr_fedora_llvm_team_llvm_compat_packages", "name": "Additional repo copr_fedora_llvm_team_llvm_compat_packages" }, { "baseurl": "https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/ppc64le/", "id": "https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le", "name": "Additional repo https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le" }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20251209/centos-stream-9-ppc64le/", "id": "copr_fedora_llvm_team_llvm_snapshots_big_merge_20251209", "module_hotfixes": true, "name": "Additional repo copr_fedora_llvm_team_llvm_snapshots_big_merge_20251209" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": 1000, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/clang-fedora-centos-testing", "result_dir": "09896379-thunderbird", "built_packages": "", "id": 9896379, "mockchain_macros": { "copr_username": "@fedora-llvm-team", "copr_projectname": "clang-fedora-centos-testing", "vendor": "Fedora Project COPR (@fedora-llvm-team/clang-fedora-centos-testing)" } } ] } [2025-12-10 22:20:05,683][ INFO][PID:1529662] Sending fedora-messaging bus message in build.end [2025-12-10 22:20:05,709][ INFO][PID:1529662] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/builder-live.log by gzip [2025-12-10 22:20:05,710][ INFO][PID:1529662] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/builder-live.log' as PID 1594132 [2025-12-10 22:20:05,748][ INFO][PID:1529662] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/builder-live.log) [2025-12-10 22:20:05,749][ INFO][PID:1529662] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/backend.log by gzip [2025-12-10 22:20:05,750][ INFO][PID:1529662] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/clang-fedora-centos-testing/centos-stream-9-ppc64le/09896379-thunderbird/backend.log' as PID 1594136