[2026-01-08 23:16:04,597][ INFO][PID:2400496] Marking build as starting [2026-01-08 23:16:04,668][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:04,670][ INFO][PID:2400496] VM allocation process starts [2026-01-08 23:16:04,706][ INFO][PID:2400496] Trying to allocate VM: ResallocHost, ticket_id=723142, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-08 23:16:07,764][ INFO][PID:2400496] Allocated host ResallocHost, ticket_id=723142, hostname=98.81.67.67, name=aws_x86_64_spot_prod_01040051_20260108_230613, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-08 23:16:08,011][ INFO][PID:2400496] Allocating ssh connection to builder [2026-01-08 23:16:08,012][ INFO][PID:2400496] Checking that builder machine is OK [2026-01-08 23:16:08,330][ INFO][PID:2400496] Running remote command: copr-builder-ready srpm-builds [2026-01-08 23:16:08,537][ INFO][PID:2400496] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-01-08 23:16:08,537][ INFO][PID:2400496] Filling build.info file with builder info [2026-01-08 23:16:08,538][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:08,538][ INFO][PID:2400496] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "ryanabx", "project_name": "cosmic-epoch", "project_dirname": "cosmic-epoch", "submitter": "ryanabx", "ended_on": null, "started_on": 1767914168.5388646, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "9988368", "build_id": 9988368, "package_name": "pop-launcher", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash -xe\\n\\nexport NAME=pop-launcher\\n\\nSCRIPT=cosmic-packaging-bootstrap.py\\nRPM_REPO_NAME=cosmic-packaging\\nRPM_REPO=https://pagure.io/fedora-cosmic/$RPM_REPO_NAME.git\\n\\ngit clone $RPM_REPO\\n\\npython3 $RPM_REPO_NAME/scripts/$SCRIPT $NAME --input $(pwd)/cosmic-packaging --output $(pwd)\\n\\nrm -rf $RPM_REPO_NAME\\n\\nls -a\\n\\ncat $NAME.spec\\n\", \"chroot\": \"fedora-41-x86_64\", \"builddeps\": \"git cargo python3\", \"resultdir\": \"\", \"repos\": \"\", \"fedora_review\": true}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "ryanabx/cosmic-epoch--ryanabx", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/ryanabx/cosmic-epoch", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch", "result_dir": "09988368", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 9988368, "mockchain_macros": { "copr_username": "ryanabx", "copr_projectname": "cosmic-epoch", "vendor": "Fedora Project COPR (ryanabx/cosmic-epoch)" } } ] } [2026-01-08 23:16:10,832][ INFO][PID:2400496] Sending fedora-messaging bus message in build.start [2026-01-08 23:16:11,438][ INFO][PID:2400496] Sending fedora-messaging bus message in chroot.start [2026-01-08 23:16:11,466][ INFO][PID:2400496] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9988368 --detached [2026-01-08 23:16:11,793][ INFO][PID:2400496] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '98.81.67.67' (ED25519) to the list of known hosts. [2026-01-08 23:16:11,794][ INFO][PID:2400496] Downloading the builder-live.log file, attempt 1 [2026-01-08 23:16:11,796][ INFO][PID:2400496] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.81.67.67 copr-rpmbuild-log [2026-01-08 23:16:16,650][ ERROR][PID:2400496] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-08 23:16:16,803][ INFO][PID:2400496] Periodic builder liveness probe: alive [2026-01-08 23:16:16,803][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:21,804][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:26,651][WARNING][PID:2400496] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-08 23:16:26,805][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:31,807][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:36,652][ INFO][PID:2400496] Downloading the builder-live.log file, attempt 2 [2026-01-08 23:16:36,655][ INFO][PID:2400496] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.81.67.67 copr-rpmbuild-log [2026-01-08 23:16:36,809][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:41,810][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:46,811][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:51,812][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:16:56,815][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:01,817][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:06,818][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:11,819][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:16,821][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:21,823][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:26,824][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:31,831][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:36,721][ ERROR][PID:2400496] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-08 23:17:36,840][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:41,845][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:46,732][WARNING][PID:2400496] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-08 23:17:46,850][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:51,855][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:17:56,736][ INFO][PID:2400496] Downloading the builder-live.log file, attempt 3 [2026-01-08 23:17:56,739][ INFO][PID:2400496] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.81.67.67 copr-rpmbuild-log [2026-01-08 23:17:56,856][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:01,858][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:06,859][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:11,860][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:16,861][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:21,863][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:26,864][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:31,868][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:36,869][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:41,870][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:46,871][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:51,880][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:18:56,806][ ERROR][PID:2400496] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-08 23:18:56,882][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:01,884][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:06,822][WARNING][PID:2400496] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-08 23:19:06,885][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:11,887][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:16,823][ INFO][PID:2400496] Downloading the builder-live.log file, attempt 4 [2026-01-08 23:19:16,825][ INFO][PID:2400496] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.81.67.67 copr-rpmbuild-log [2026-01-08 23:19:16,888][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:21,891][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:26,893][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:31,897][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:36,898][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:41,899][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:46,900][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:51,901][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:19:56,902][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:01,903][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:06,904][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:11,905][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:16,851][ ERROR][PID:2400496] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-08 23:20:16,906][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:21,907][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:26,856][ ERROR][PID:2400496] Giving up for unstable SSH, failures: 1767914186.6516948, 1767914266.7327738, 1767914346.8227081, 1767914426.8566582 [2026-01-08 23:20:26,857][ ERROR][PID:2400496] Re-try request for task on 'ResallocHost, ticket_id=723142, hostname=98.81.67.67, name=aws_x86_64_spot_prod_01040051_20260108_230613, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-08 23:20:26,858][ INFO][PID:2400496] Releasing VM back to pool [2026-01-08 23:20:26,868][ INFO][PID:2400496] Retry #1 (on other host) [2026-01-08 23:20:26,869][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:26,869][ INFO][PID:2400496] VM allocation process starts [2026-01-08 23:20:26,886][ INFO][PID:2400496] Trying to allocate VM: ResallocHost, ticket_id=723185, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-08 23:20:29,904][ INFO][PID:2400496] Allocated host ResallocHost, ticket_id=723185, hostname=3.238.245.76, name=aws_x86_64_normalreserved_prod_01040164_20260108_231706, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-08 23:20:29,906][ INFO][PID:2400496] Allocating ssh connection to builder [2026-01-08 23:20:29,906][ INFO][PID:2400496] Checking that builder machine is OK [2026-01-08 23:20:30,238][ INFO][PID:2400496] Running remote command: copr-builder-ready srpm-builds [2026-01-08 23:20:30,435][ INFO][PID:2400496] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-01-08 23:20:30,436][ INFO][PID:2400496] Filling build.info file with builder info [2026-01-08 23:20:30,447][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:30,448][ INFO][PID:2400496] Not re-notifying FE and msg buses for the new host. [2026-01-08 23:20:30,449][ INFO][PID:2400496] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9988368 --detached [2026-01-08 23:20:30,778][ INFO][PID:2400496] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.238.245.76' (ED25519) to the list of known hosts. [2026-01-08 23:20:30,779][ INFO][PID:2400496] Downloading the builder-live.log file, attempt 1 [2026-01-08 23:20:30,781][ INFO][PID:2400496] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.245.76 copr-rpmbuild-log [2026-01-08 23:20:35,780][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:40,781][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:45,782][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:50,783][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:20:55,784][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:21:00,785][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:21:05,786][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:21:10,787][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:21:15,788][ INFO][PID:2400496] Checking for cancel request [2026-01-08 23:21:18,953][ INFO][PID:2400496] Downloading results from builder [2026-01-08 23:21:19,169][ INFO][PID:2400496] rsyncing of mockbuilder@3.238.245.76:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368 started [2026-01-08 23:21:19,170][ INFO][PID:2400496] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.238.245.76:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/ &> /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/build-09988368.rsync.log [2026-01-08 23:21:19,846][ INFO][PID:2400496] rsyncing finished. [2026-01-08 23:21:19,847][ INFO][PID:2400496] Releasing VM back to pool [2026-01-08 23:21:19,856][ INFO][PID:2400496] Searching for 'success' file in resultdir [2026-01-08 23:21:19,856][ INFO][PID:2400496] Getting build details [2026-01-08 23:21:19,857][ INFO][PID:2400496] Retrieving SRPM info from /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368 [2026-01-08 23:21:19,858][ INFO][PID:2400496] SRPM URL: https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch/srpm-builds/09988368/pop-launcher-1.2.7^git20251001.eead361-1.src.rpm [2026-01-08 23:21:19,858][ INFO][PID:2400496] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch/srpm-builds/09988368/pop-launcher-1.2.7^git20251001.eead361-1.src.rpm', 'pkg_name': 'pop-launcher', 'pkg_version': '1:1.2.7^git20251001.eead361-1'} [2026-01-08 23:21:19,859][ INFO][PID:2400496] Finished build: id=9988368 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/ryanabx/cosmic-epoch chroot=srpm-builds [2026-01-08 23:21:19,860][ INFO][PID:2400496] Worker succeeded build, took 49.41208076477051 [2026-01-08 23:21:19,861][ INFO][PID:2400496] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "ryanabx", "project_name": "cosmic-epoch", "project_dirname": "cosmic-epoch", "submitter": "ryanabx", "ended_on": 1767914479.8607845, "started_on": 1767914430.4487038, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "9988368", "build_id": 9988368, "package_name": "pop-launcher", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash -xe\\n\\nexport NAME=pop-launcher\\n\\nSCRIPT=cosmic-packaging-bootstrap.py\\nRPM_REPO_NAME=cosmic-packaging\\nRPM_REPO=https://pagure.io/fedora-cosmic/$RPM_REPO_NAME.git\\n\\ngit clone $RPM_REPO\\n\\npython3 $RPM_REPO_NAME/scripts/$SCRIPT $NAME --input $(pwd)/cosmic-packaging --output $(pwd)\\n\\nrm -rf $RPM_REPO_NAME\\n\\nls -a\\n\\ncat $NAME.spec\\n\", \"chroot\": \"fedora-41-x86_64\", \"builddeps\": \"git cargo python3\", \"resultdir\": \"\", \"repos\": \"\", \"fedora_review\": true}", "pkg_name": "pop-launcher", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch/srpm-builds/09988368/pop-launcher-1.2.7^git20251001.eead361-1.src.rpm", "uses_devel_repo": null, "sandbox": "ryanabx/cosmic-epoch--ryanabx", "results": { "name": "pop-launcher", "epoch": 1, "version": "1.2.7^git20251001.eead361", "release": "1", "exclusivearch": [], "excludearch": [ "i386", "i486", "i586", "i686", "pentium3", "pentium4", "athlon", "geode" ] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/ryanabx/cosmic-epoch", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch", "result_dir": "09988368", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "1:1.2.7^git20251001.eead361-1", "id": 9988368, "mockchain_macros": { "copr_username": "ryanabx", "copr_projectname": "cosmic-epoch", "vendor": "Fedora Project COPR (ryanabx/cosmic-epoch)" } } ] } [2026-01-08 23:21:19,926][ INFO][PID:2400496] Sending fedora-messaging bus message in build.end [2026-01-08 23:21:19,952][ INFO][PID:2400496] Compressing /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/builder-live.log by gzip [2026-01-08 23:21:19,953][ INFO][PID:2400496] Running command 'gzip /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/builder-live.log' as PID 2427004 [2026-01-08 23:21:19,960][ INFO][PID:2400496] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/builder-live.log) [2026-01-08 23:21:19,961][ INFO][PID:2400496] Compressing /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/backend.log by gzip [2026-01-08 23:21:19,962][ INFO][PID:2400496] Running command 'gzip /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/srpm-builds/09988368/backend.log' as PID 2427007