[2026-01-13 17:08:44,230][ INFO][PID:415991] Marking build as starting [2026-01-13 17:08:44,348][ INFO][PID:415991] Checking that /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/repodata/repomd.xml exists [2026-01-13 17:08:44,365][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:08:44,367][ INFO][PID:415991] VM allocation process starts [2026-01-13 17:08:44,419][ INFO][PID:415991] Trying to allocate VM: ResallocHost, ticket_id=796397, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 17:08:49,426][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:08:50,491][ INFO][PID:415991] Allocated host ResallocHost, ticket_id=796397, hostname=44.201.184.213, name=aws_x86_64_normal_prod_01203798_20260113_170702, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 17:08:50,499][ INFO][PID:415991] Allocating ssh connection to builder [2026-01-13 17:08:50,501][ INFO][PID:415991] Checking that builder machine is OK [2026-01-13 17:08:50,889][ INFO][PID:415991] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-01-13 17:08:51,191][ INFO][PID:415991] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-01-13 17:08:51,193][ INFO][PID:415991] Filling build.info file with builder info [2026-01-13 17:08:51,195][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:08:51,196][ INFO][PID:415991] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@oisf", "project_name": "suricata-7.0", "project_dirname": "suricata-7.0", "submitter": "jasonish", "ended_on": null, "started_on": 1768324131.1961443, "submitted_on": null, "status": 3, "chroot": "fedora-42-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10004499-fedora-42-x86_64", "build_id": 10004499, "package_name": "suricata", "package_version": "1:7.0.14-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@oisf/suricata-7.0/suricata", "git_hash": "4316cbfb701da56cc29400f2055e80e0503d36e7", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@oisf/suricata-7.0--jasonish", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@oisf/suricata-7.0/fedora-42-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@oisf/suricata-7.0", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@oisf/suricata-7.0", "result_dir": "10004499-suricata", "built_packages": "", "id": 10004499, "mockchain_macros": { "copr_username": "@oisf", "copr_projectname": "suricata-7.0", "vendor": "Fedora Project COPR (@oisf/suricata-7.0)" } } ] } [2026-01-13 17:08:51,350][ INFO][PID:415991] Sending fedora-messaging bus message in build.start [2026-01-13 17:08:52,575][ INFO][PID:415991] Sending fedora-messaging bus message in chroot.start [2026-01-13 17:08:52,659][ INFO][PID:415991] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10004499-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-01-13 17:08:53,064][ INFO][PID:415991] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.201.184.213' (ED25519) to the list of known hosts. [2026-01-13 17:08:53,068][ INFO][PID:415991] Downloading the builder-live.log file, attempt 1 [2026-01-13 17:08:53,071][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.184.213 copr-rpmbuild-log [2026-01-13 17:08:58,107][ INFO][PID:415991] Periodic builder liveness probe: alive [2026-01-13 17:08:58,112][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:03,114][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:08,116][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:13,119][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:18,131][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:23,132][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:28,133][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:33,136][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:38,143][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:43,147][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:48,155][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:53,156][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:09:58,161][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:03,163][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:08,166][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:13,173][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:18,178][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:23,181][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:28,188][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:33,197][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:38,201][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:43,210][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:48,221][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:53,241][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:10:58,245][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:03,247][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:08,251][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:13,258][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:18,265][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:23,272][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:28,280][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:33,282][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:38,287][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:43,290][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:48,295][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:53,309][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:11:58,312][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:03,316][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:08,320][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:13,323][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:18,325][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:23,329][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:28,331][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:33,336][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:38,340][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:43,341][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:48,344][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:53,351][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:12:58,355][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:03,357][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:08,359][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:13,365][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:18,367][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:23,371][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:25,414][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:13:28,375][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:33,381][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:35,424][WARNING][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:13:38,384][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:43,386][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:45,424][ INFO][PID:415991] Downloading the builder-live.log file, attempt 2 [2026-01-13 17:13:45,445][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.184.213 copr-rpmbuild-log [2026-01-13 17:13:48,391][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:53,396][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:13:58,401][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:03,404][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:08,408][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:13,410][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:18,414][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:23,418][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:28,419][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:33,423][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:38,480][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:43,486][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:45,533][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:14:48,487][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:53,492][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:14:55,551][WARNING][PID:415991] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:14:58,495][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:03,501][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:05,554][ INFO][PID:415991] Downloading the builder-live.log file, attempt 3 [2026-01-13 17:15:05,562][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.184.213 copr-rpmbuild-log [2026-01-13 17:15:08,508][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:13,513][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:18,519][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:23,520][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:28,521][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:33,524][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:38,528][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:43,545][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:48,553][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:53,564][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:15:58,569][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:03,573][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:05,645][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:16:08,582][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:13,594][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:15,659][WARNING][PID:415991] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:16:18,610][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:23,615][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:25,661][ INFO][PID:415991] Downloading the builder-live.log file, attempt 4 [2026-01-13 17:16:25,669][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.184.213 copr-rpmbuild-log [2026-01-13 17:16:28,627][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:33,630][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:38,639][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:43,653][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:48,654][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:53,658][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:16:58,662][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:03,664][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:08,667][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:13,676][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:18,678][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:23,680][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:25,758][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:17:28,690][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:33,691][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:35,775][ ERROR][PID:415991] Giving up for unstable SSH, failures: 1768324415.4240444, 1768324495.5510247, 1768324575.6590402, 1768324655.774721 [2026-01-13 17:17:35,778][ ERROR][PID:415991] Re-try request for task on 'ResallocHost, ticket_id=796397, hostname=44.201.184.213, name=aws_x86_64_normal_prod_01203798_20260113_170702, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-13 17:17:35,780][ INFO][PID:415991] Releasing VM back to pool [2026-01-13 17:17:35,803][ INFO][PID:415991] Retry #1 (on other host) [2026-01-13 17:17:35,805][ INFO][PID:415991] Checking that /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/repodata/repomd.xml exists [2026-01-13 17:17:35,810][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:35,812][ INFO][PID:415991] VM allocation process starts [2026-01-13 17:17:35,836][ INFO][PID:415991] Trying to allocate VM: ResallocHost, ticket_id=796936, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 17:17:40,842][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:45,844][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:50,845][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:17:55,847][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:00,851][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:05,853][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:10,858][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:15,865][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:20,870][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:25,877][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:30,888][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:35,893][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:40,908][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:45,909][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:50,918][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:18:55,935][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:00,938][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:05,939][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:10,943][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:15,949][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:20,953][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:24,172][ INFO][PID:415991] Allocated host ResallocHost, ticket_id=796936, hostname=50.16.17.147, name=aws_x86_64_spot_prod_01204387_20260113_171718, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 17:19:24,172][ INFO][PID:415991] Allocating ssh connection to builder [2026-01-13 17:19:24,175][ INFO][PID:415991] Checking that builder machine is OK [2026-01-13 17:19:24,653][ INFO][PID:415991] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-01-13 17:19:24,952][ INFO][PID:415991] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-01-13 17:19:24,955][ INFO][PID:415991] Filling build.info file with builder info [2026-01-13 17:19:24,987][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:24,991][ INFO][PID:415991] Not re-notifying FE and msg buses for the new host. [2026-01-13 17:19:24,995][ INFO][PID:415991] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10004499-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-01-13 17:19:25,463][ INFO][PID:415991] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '50.16.17.147' (ED25519) to the list of known hosts. [2026-01-13 17:19:25,466][ INFO][PID:415991] Downloading the builder-live.log file, attempt 1 [2026-01-13 17:19:25,473][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.17.147 copr-rpmbuild-log [2026-01-13 17:19:30,497][ INFO][PID:415991] Periodic builder liveness probe: alive [2026-01-13 17:19:30,502][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:35,511][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:40,512][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:45,515][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:50,520][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:19:55,524][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:00,528][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:05,532][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:10,534][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:15,536][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:20,543][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:25,544][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:30,546][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:35,550][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:40,552][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:45,558][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:50,562][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:20:55,566][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:00,570][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:05,574][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:10,578][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:15,579][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:20,581][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:25,583][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:30,584][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:31,638][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:21:35,586][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:40,588][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:41,639][WARNING][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:21:45,589][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:50,591][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:21:51,641][ INFO][PID:415991] Downloading the builder-live.log file, attempt 2 [2026-01-13 17:21:51,642][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.17.147 copr-rpmbuild-log [2026-01-13 17:21:55,592][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:00,594][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:05,595][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:10,606][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:15,607][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:20,609][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:25,610][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:30,610][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:35,843][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:40,844][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:45,845][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:50,846][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:22:51,709][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:22:55,847][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:00,850][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:01,712][WARNING][PID:415991] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:23:05,852][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:10,853][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:11,713][ INFO][PID:415991] Downloading the builder-live.log file, attempt 3 [2026-01-13 17:23:11,714][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.17.147 copr-rpmbuild-log [2026-01-13 17:23:15,856][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:20,867][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:25,868][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:30,869][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:35,871][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:40,872][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:45,873][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:50,874][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:23:55,875][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:00,876][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:05,877][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:10,878][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:11,770][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:24:15,879][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:20,880][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:21,788][WARNING][PID:415991] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:24:25,881][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:30,882][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:31,789][ INFO][PID:415991] Downloading the builder-live.log file, attempt 4 [2026-01-13 17:24:31,790][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.17.147 copr-rpmbuild-log [2026-01-13 17:24:35,883][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:40,884][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:45,885][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:50,886][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:24:55,887][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:00,888][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:05,889][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:10,890][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:15,891][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:20,892][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:25,894][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:30,896][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:31,858][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:25:35,897][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:40,898][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:41,863][ ERROR][PID:415991] Giving up for unstable SSH, failures: 1768324901.6390846, 1768324981.7125468, 1768325061.7886333, 1768325141.8629243 [2026-01-13 17:25:41,863][ ERROR][PID:415991] Re-try request for task on 'ResallocHost, ticket_id=796936, hostname=50.16.17.147, name=aws_x86_64_spot_prod_01204387_20260113_171718, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-13 17:25:41,864][ INFO][PID:415991] Releasing VM back to pool [2026-01-13 17:25:41,874][ INFO][PID:415991] Retry #2 (on other host) [2026-01-13 17:25:41,875][ INFO][PID:415991] Checking that /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/repodata/repomd.xml exists [2026-01-13 17:25:41,875][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:41,876][ INFO][PID:415991] VM allocation process starts [2026-01-13 17:25:41,888][ INFO][PID:415991] Trying to allocate VM: ResallocHost, ticket_id=797099, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 17:25:44,900][ INFO][PID:415991] Allocated host ResallocHost, ticket_id=797099, hostname=54.197.71.162, name=aws_x86_64_spot_prod_01204600_20260113_172124, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-13 17:25:44,900][ INFO][PID:415991] Allocating ssh connection to builder [2026-01-13 17:25:44,900][ INFO][PID:415991] Checking that builder machine is OK [2026-01-13 17:25:45,241][ INFO][PID:415991] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-01-13 17:25:45,514][ INFO][PID:415991] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-01-13 17:25:45,514][ INFO][PID:415991] Filling build.info file with builder info [2026-01-13 17:25:45,523][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:45,525][ INFO][PID:415991] Not re-notifying FE and msg buses for the new host. [2026-01-13 17:25:45,526][ INFO][PID:415991] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10004499-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-01-13 17:25:45,904][ INFO][PID:415991] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.197.71.162' (ED25519) to the list of known hosts. [2026-01-13 17:25:45,905][ INFO][PID:415991] Downloading the builder-live.log file, attempt 1 [2026-01-13 17:25:45,908][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.197.71.162 copr-rpmbuild-log [2026-01-13 17:25:50,906][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:25:55,909][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:00,911][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:05,913][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:10,919][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:15,921][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:20,959][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:25,963][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:30,966][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:35,967][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:40,968][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:45,976][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:51,090][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:26:56,228][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:01,230][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:06,232][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:11,233][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:16,234][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:21,235][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:26,236][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:31,251][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:36,252][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:41,253][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:46,254][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:51,256][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:27:56,257][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:01,258][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:06,311][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:11,312][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:16,314][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:21,315][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:26,316][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:31,323][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:36,333][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:41,334][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:46,335][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:51,337][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:28:56,340][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:01,341][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:06,342][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:11,344][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:16,353][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:21,354][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:26,355][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:31,363][ INFO][PID:415991] Periodic builder liveness probe: alive [2026-01-13 17:29:31,364][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:36,365][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:41,367][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:46,368][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:51,385][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:29:56,851][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:01,855][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:06,858][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:11,859][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:16,861][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:21,864][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:26,868][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:31,869][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:36,877][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:41,882][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:46,883][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:51,887][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:30:56,894][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:01,902][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:06,905][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:11,907][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:16,911][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:21,914][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:26,921][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:31,943][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:36,947][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:41,952][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:46,954][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:51,959][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:31:56,961][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:01,974][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:06,985][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:11,995][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:17,004][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:22,007][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:26,409][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:32:27,015][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:32,017][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:36,411][WARNING][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:32:37,019][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:42,023][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:46,413][ INFO][PID:415991] Downloading the builder-live.log file, attempt 2 [2026-01-13 17:32:46,417][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.197.71.162 copr-rpmbuild-log [2026-01-13 17:32:47,035][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:52,037][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:32:57,038][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:02,040][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:07,043][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:12,044][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:17,046][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:22,049][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:27,052][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:32,053][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:37,056][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:42,058][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:46,487][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:33:47,064][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:52,065][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:33:56,490][WARNING][PID:415991] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:33:57,069][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:02,070][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:06,491][ INFO][PID:415991] Downloading the builder-live.log file, attempt 3 [2026-01-13 17:34:06,492][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.197.71.162 copr-rpmbuild-log [2026-01-13 17:34:07,071][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:12,072][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:17,073][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:22,074][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:27,076][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:32,077][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:37,078][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:42,079][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:47,080][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:52,081][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:34:57,081][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:02,082][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:06,560][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:35:07,497][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:12,498][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:16,566][WARNING][PID:415991] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-13 17:35:17,499][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:22,500][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:26,567][ INFO][PID:415991] Downloading the builder-live.log file, attempt 4 [2026-01-13 17:35:26,570][ INFO][PID:415991] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.197.71.162 copr-rpmbuild-log [2026-01-13 17:35:27,501][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:32,508][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:37,516][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:42,517][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:47,519][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:52,521][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:35:57,522][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:02,523][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:07,524][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:12,526][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:17,530][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:22,532][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:26,641][ ERROR][PID:415991] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-13 17:36:27,537][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:32,540][ INFO][PID:415991] Checking for cancel request [2026-01-13 17:36:36,654][ ERROR][PID:415991] Giving up for unstable SSH, failures: 1768325556.4118888, 1768325636.4907792, 1768325716.5666971, 1768325796.654662 [2026-01-13 17:36:36,655][ ERROR][PID:415991] Re-try request for task on 'ResallocHost, ticket_id=797099, hostname=54.197.71.162, name=aws_x86_64_spot_prod_01204600_20260113_172124, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-13 17:36:36,656][ INFO][PID:415991] Releasing VM back to pool [2026-01-13 17:36:36,665][ ERROR][PID:415991] Backend process error: Three host tried without success: {'44.201.184.213', '54.197.71.162', '50.16.17.147'} [2026-01-13 17:36:36,665][WARNING][PID:415991] Switching not-finished job state to 'failed' [2026-01-13 17:36:36,666][ INFO][PID:415991] Worker failed build, took 651.1398341655731 [2026-01-13 17:36:36,666][ INFO][PID:415991] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@oisf", "project_name": "suricata-7.0", "project_dirname": "suricata-7.0", "submitter": "jasonish", "ended_on": 1768325796.6656778, "started_on": 1768325145.5258436, "submitted_on": null, "status": 0, "chroot": "fedora-42-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10004499-fedora-42-x86_64", "build_id": 10004499, "package_name": "suricata", "package_version": "1:7.0.14-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@oisf/suricata-7.0/suricata", "git_hash": "4316cbfb701da56cc29400f2055e80e0503d36e7", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@oisf/suricata-7.0--jasonish", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@oisf/suricata-7.0/fedora-42-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@oisf/suricata-7.0", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@oisf/suricata-7.0", "result_dir": "10004499-suricata", "built_packages": "", "id": 10004499, "mockchain_macros": { "copr_username": "@oisf", "copr_projectname": "suricata-7.0", "vendor": "Fedora Project COPR (@oisf/suricata-7.0)" } } ] } [2026-01-13 17:36:36,722][ INFO][PID:415991] Sending fedora-messaging bus message in build.end [2026-01-13 17:36:36,760][ INFO][PID:415991] Compressing /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/10004499-suricata/builder-live.log by gzip [2026-01-13 17:36:36,761][ INFO][PID:415991] Running command 'gzip /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/10004499-suricata/builder-live.log' as PID 677042 [2026-01-13 17:36:36,763][ INFO][PID:415991] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/10004499-suricata/builder-live.log) [2026-01-13 17:36:36,763][ INFO][PID:415991] Compressing /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/10004499-suricata/backend.log by gzip [2026-01-13 17:36:36,765][ INFO][PID:415991] Running command 'gzip /var/lib/copr/public_html/results/@oisf/suricata-7.0/fedora-42-x86_64/10004499-suricata/backend.log' as PID 677043