[2026-01-14 14:50:48,526][ INFO][PID:2773878] Marking build as starting [2026-01-14 14:50:48,599][ INFO][PID:2773878] Checking that /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/repodata/repomd.xml exists [2026-01-14 14:50:48,713][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:50:48,715][ INFO][PID:2773878] VM allocation process starts [2026-01-14 14:50:48,750][ INFO][PID:2773878] Trying to allocate VM: ResallocHost, ticket_id=811255, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-14 14:50:51,780][ INFO][PID:2773878] Allocated host ResallocHost, ticket_id=811255, hostname=18.207.130.110, name=aws_x86_64_spot_prod_01230190_20260114_144700, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-14 14:50:51,781][ INFO][PID:2773878] Allocating ssh connection to builder [2026-01-14 14:50:51,782][ INFO][PID:2773878] Checking that builder machine is OK [2026-01-14 14:50:52,117][ INFO][PID:2773878] Running remote command: copr-builder-ready epel-9-x86_64 [2026-01-14 14:50:52,329][ INFO][PID:2773878] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-14 14:50:52,330][ INFO][PID:2773878] Filling build.info file with builder info [2026-01-14 14:50:52,330][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:50:52,333][ INFO][PID:2773878] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@oamg", "project_name": "leapp", "project_dirname": "leapp", "submitter": "packit", "ended_on": null, "started_on": 1768402252.3331587, "submitted_on": null, "status": 3, "chroot": "epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10007880-epel-9-x86_64", "build_id": 10007880, "package_name": "leapp-repository", "package_version": "0.23.0-0.20260114145007071811.pr1475.115.g87a22ce4", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@oamg/leapp/leapp-repository", "git_hash": "ab822a2a4315f3274481d8ae21b73350849c368a", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@oamg/leapp--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@oamg/leapp/epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@oamg/leapp", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@oamg/leapp", "result_dir": "10007880-leapp-repository", "built_packages": "", "id": 10007880, "mockchain_macros": { "copr_username": "@oamg", "copr_projectname": "leapp", "vendor": "Fedora Project COPR (@oamg/leapp)" } } ] } [2026-01-14 14:50:52,407][ INFO][PID:2773878] Sending fedora-messaging bus message in build.start [2026-01-14 14:50:53,269][ INFO][PID:2773878] Sending fedora-messaging bus message in chroot.start [2026-01-14 14:50:53,319][ INFO][PID:2773878] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10007880-epel-9-x86_64 --chroot epel-9-x86_64 --detached [2026-01-14 14:50:53,687][ INFO][PID:2773878] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '18.207.130.110' (ED25519) to the list of known hosts. [2026-01-14 14:50:53,688][ INFO][PID:2773878] Downloading the builder-live.log file, attempt 1 [2026-01-14 14:50:53,693][ INFO][PID:2773878] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.207.130.110 copr-rpmbuild-log [2026-01-14 14:50:58,700][ INFO][PID:2773878] Periodic builder liveness probe: alive [2026-01-14 14:50:58,701][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:02,007][ ERROR][PID:2773878] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-14 14:51:03,702][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:08,704][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:12,013][WARNING][PID:2773878] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-14 14:51:13,709][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:18,710][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:22,015][ INFO][PID:2773878] Downloading the builder-live.log file, attempt 2 [2026-01-14 14:51:22,016][ INFO][PID:2773878] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.207.130.110 copr-rpmbuild-log [2026-01-14 14:51:23,712][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:28,715][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:33,717][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:38,719][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:43,722][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:48,724][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:53,726][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:51:58,729][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:03,730][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:08,731][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:13,732][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:18,734][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:22,074][ ERROR][PID:2773878] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-14 14:52:23,735][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:28,737][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:32,101][WARNING][PID:2773878] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-14 14:52:33,737][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:38,738][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:42,102][ INFO][PID:2773878] Downloading the builder-live.log file, attempt 3 [2026-01-14 14:52:42,103][ INFO][PID:2773878] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.207.130.110 copr-rpmbuild-log [2026-01-14 14:52:43,739][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:48,740][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:53,742][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:52:58,743][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:03,744][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:08,745][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:13,746][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:18,773][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:23,775][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:28,776][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:33,777][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:38,778][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:42,170][ ERROR][PID:2773878] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-14 14:53:43,779][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:48,780][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:52,176][WARNING][PID:2773878] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-14 14:53:53,781][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:53:58,782][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:02,177][ INFO][PID:2773878] Downloading the builder-live.log file, attempt 4 [2026-01-14 14:54:02,178][ INFO][PID:2773878] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.207.130.110 copr-rpmbuild-log [2026-01-14 14:54:03,783][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:08,784][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:13,785][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:18,787][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:23,788][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:28,789][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:33,790][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:38,791][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:43,792][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:48,793][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:53,794][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:54:58,795][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:02,245][ ERROR][PID:2773878] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-14 14:55:03,796][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:08,797][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:12,272][ ERROR][PID:2773878] Giving up for unstable SSH, failures: 1768402272.0134284, 1768402352.101876, 1768402432.1768234, 1768402512.2719605 [2026-01-14 14:55:12,273][ ERROR][PID:2773878] Re-try request for task on 'ResallocHost, ticket_id=811255, hostname=18.207.130.110, name=aws_x86_64_spot_prod_01230190_20260114_144700, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-14 14:55:12,273][ INFO][PID:2773878] Releasing VM back to pool [2026-01-14 14:55:12,284][ INFO][PID:2773878] Retry #1 (on other host) [2026-01-14 14:55:12,284][ INFO][PID:2773878] Checking that /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/repodata/repomd.xml exists [2026-01-14 14:55:12,285][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:12,285][ INFO][PID:2773878] VM allocation process starts [2026-01-14 14:55:12,296][ INFO][PID:2773878] Trying to allocate VM: ResallocHost, ticket_id=811283, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-14 14:55:15,318][ INFO][PID:2773878] Allocated host ResallocHost, ticket_id=811283, hostname=100.30.182.198, name=aws_x86_64_normalreserved_prod_01230395_20260114_145016, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-14 14:55:15,319][ INFO][PID:2773878] Allocating ssh connection to builder [2026-01-14 14:55:15,319][ INFO][PID:2773878] Checking that builder machine is OK [2026-01-14 14:55:15,668][ INFO][PID:2773878] Running remote command: copr-builder-ready epel-9-x86_64 [2026-01-14 14:55:15,894][ INFO][PID:2773878] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-14 14:55:15,895][ INFO][PID:2773878] Filling build.info file with builder info [2026-01-14 14:55:15,915][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:15,918][ INFO][PID:2773878] Not re-notifying FE and msg buses for the new host. [2026-01-14 14:55:15,921][ INFO][PID:2773878] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10007880-epel-9-x86_64 --chroot epel-9-x86_64 --detached [2026-01-14 14:55:16,272][ INFO][PID:2773878] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '100.30.182.198' (ED25519) to the list of known hosts. [2026-01-14 14:55:16,272][ INFO][PID:2773878] Downloading the builder-live.log file, attempt 1 [2026-01-14 14:55:16,274][ INFO][PID:2773878] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.30.182.198 copr-rpmbuild-log [2026-01-14 14:55:21,273][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:26,274][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:31,328][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:36,328][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:41,329][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:46,330][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:51,331][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:55:56,332][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:56:01,337][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:56:06,338][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:56:11,339][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:56:16,340][ INFO][PID:2773878] Checking for cancel request [2026-01-14 14:56:20,883][ INFO][PID:2773878] Downloading results from builder [2026-01-14 14:56:20,884][ INFO][PID:2773878] rsyncing of mockbuilder@100.30.182.198:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository started [2026-01-14 14:56:20,885][ INFO][PID:2773878] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@100.30.182.198:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/ &> /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/build-10007880.rsync.log [2026-01-14 14:56:21,101][ INFO][PID:2773878] rsyncing finished. [2026-01-14 14:56:21,102][ INFO][PID:2773878] Releasing VM back to pool [2026-01-14 14:56:21,110][ INFO][PID:2773878] Searching for 'success' file in resultdir [2026-01-14 14:56:21,111][ ERROR][PID:2773878] Build failed: Backend process error: No success file => build failure [2026-01-14 14:56:21,111][ INFO][PID:2773878] Finished build: id=10007880 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@oamg/leapp chroot=epel-9-x86_64 [2026-01-14 14:56:21,113][ ERROR][PID:2773878] Backend process error: No success file => build failure [2026-01-14 14:56:21,113][ INFO][PID:2773878] Worker failed build, took 65.19511651992798 [2026-01-14 14:56:21,114][ INFO][PID:2773878] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@oamg", "project_name": "leapp", "project_dirname": "leapp", "submitter": "packit", "ended_on": 1768402581.113747, "started_on": 1768402515.9186304, "submitted_on": null, "status": 0, "chroot": "epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10007880-epel-9-x86_64", "build_id": 10007880, "package_name": "leapp-repository", "package_version": "0.23.0-0.20260114145007071811.pr1475.115.g87a22ce4", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@oamg/leapp/leapp-repository", "git_hash": "ab822a2a4315f3274481d8ae21b73350849c368a", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@oamg/leapp--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@oamg/leapp/epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@oamg/leapp", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@oamg/leapp", "result_dir": "10007880-leapp-repository", "built_packages": "", "id": 10007880, "mockchain_macros": { "copr_username": "@oamg", "copr_projectname": "leapp", "vendor": "Fedora Project COPR (@oamg/leapp)" } } ] } [2026-01-14 14:56:21,155][ INFO][PID:2773878] Sending fedora-messaging bus message in build.end [2026-01-14 14:56:21,182][ INFO][PID:2773878] Compressing /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/builder-live.log by gzip [2026-01-14 14:56:21,183][ INFO][PID:2773878] Running command 'gzip /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/builder-live.log' as PID 2812704 [2026-01-14 14:56:21,618][ INFO][PID:2773878] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/builder-live.log) [2026-01-14 14:56:21,618][ INFO][PID:2773878] Compressing /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/backend.log by gzip [2026-01-14 14:56:21,619][ INFO][PID:2773878] Running command 'gzip /var/lib/copr/public_html/results/@oamg/leapp/epel-9-x86_64/10007880-leapp-repository/backend.log' as PID 2812734