[2025-07-11 16:17:20,516][ INFO][PID:3101942] Marking build as starting [2025-07-11 16:17:20,543][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:20,544][ INFO][PID:3101942] VM allocation process starts [2025-07-11 16:17:20,563][ INFO][PID:3101942] Trying to allocate VM: ResallocHost, ticket_id=3896419, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-11 16:17:23,618][ INFO][PID:3101942] Allocated host ResallocHost, ticket_id=3896419, hostname=13.218.36.6, name=aws_x86_64_spot_prod_05296056_20250711_161417, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-11 16:17:23,619][ INFO][PID:3101942] Allocating ssh connection to builder [2025-07-11 16:17:23,620][ INFO][PID:3101942] Checking that builder machine is OK [2025-07-11 16:17:24,062][ INFO][PID:3101942] Running remote command: copr-builder-ready srpm-builds [2025-07-11 16:17:24,267][ INFO][PID:3101942] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2025-07-11 16:17:24,267][ INFO][PID:3101942] Filling build.info file with builder info [2025-07-11 16:17:24,268][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:24,269][ INFO][PID:3101942] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "packit", "project_name": "containers-skopeo-2645", "project_dirname": "containers-skopeo-2645", "submitter": "packit", "ended_on": null, "started_on": 1752250644.2690892, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "9267054", "build_id": 9267054, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources -p skopeo-fedora --result-dir \\\"$resultdir\\\" --pr-id 2645 --merge-pr --target-branch main --job-config-index 0 --merged-ref 'main' --no-create-symlinks https://github.com/containers/skopeo\\n\\n\", \"chroot\": \"fedora-42-x86_64\", \"builddeps\": \"packit make\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "packit/containers-skopeo-2645--packit", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-42-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/packit/containers-skopeo-2645", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/containers-skopeo-2645", "result_dir": "09267054", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 9267054, "mockchain_macros": { "copr_username": "packit", "copr_projectname": "containers-skopeo-2645", "vendor": "Fedora Project COPR (packit/containers-skopeo-2645)" } } ] } [2025-07-11 16:17:24,290][ INFO][PID:3101942] Sending fedora-messaging bus message in build.start [2025-07-11 16:17:25,020][ INFO][PID:3101942] Sending fedora-messaging bus message in chroot.start [2025-07-11 16:17:25,049][ INFO][PID:3101942] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9267054 --detached [2025-07-11 16:17:25,375][ INFO][PID:3101942] Downloading the builder-live.log file, attempt 1 [2025-07-11 16:17:25,377][ INFO][PID:3101942] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.218.36.6 copr-rpmbuild-log [2025-07-11 16:17:30,423][ INFO][PID:3101942] Periodic builder liveness probe: alive [2025-07-11 16:17:30,424][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:35,425][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:40,426][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:45,428][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:50,429][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:17:55,430][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:00,431][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:05,433][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:10,434][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:15,435][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:18,584][ ERROR][PID:3101942] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-11 16:18:20,436][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:25,439][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:28,586][WARNING][PID:3101942] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-11 16:18:30,445][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:35,447][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:38,587][ INFO][PID:3101942] Downloading the builder-live.log file, attempt 2 [2025-07-11 16:18:38,590][ INFO][PID:3101942] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.218.36.6 copr-rpmbuild-log [2025-07-11 16:18:40,448][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:45,449][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:50,450][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:18:55,452][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:00,453][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:05,454][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:10,455][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:15,457][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:20,458][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:25,459][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:30,461][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:35,462][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:38,659][ ERROR][PID:3101942] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-11 16:19:40,464][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:45,467][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:48,665][WARNING][PID:3101942] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-11 16:19:50,469][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:55,473][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:19:58,669][ INFO][PID:3101942] Downloading the builder-live.log file, attempt 3 [2025-07-11 16:19:58,671][ INFO][PID:3101942] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.218.36.6 copr-rpmbuild-log [2025-07-11 16:20:00,476][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:05,481][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:10,490][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:15,491][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:20,492][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:25,494][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:30,495][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:35,496][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:40,497][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:45,498][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:50,499][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:55,501][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:20:58,747][ ERROR][PID:3101942] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-11 16:21:00,502][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:05,503][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:08,752][WARNING][PID:3101942] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-11 16:21:10,504][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:15,505][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:18,753][ INFO][PID:3101942] Downloading the builder-live.log file, attempt 4 [2025-07-11 16:21:18,754][ INFO][PID:3101942] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.218.36.6 copr-rpmbuild-log [2025-07-11 16:21:20,506][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:25,507][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:30,508][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:35,510][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:40,511][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:45,512][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:50,513][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:21:55,514][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:00,515][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:05,516][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:10,517][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:15,518][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:18,822][ ERROR][PID:3101942] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-11 16:22:20,519][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:25,520][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:28,823][ ERROR][PID:3101942] Giving up for unstable SSH, failures: 1752250708.5861218, 1752250788.665031, 1752250868.752499, 1752250948.8233569 [2025-07-11 16:22:28,824][ ERROR][PID:3101942] Re-try request for task on 'ResallocHost, ticket_id=3896419, hostname=13.218.36.6, name=aws_x86_64_spot_prod_05296056_20250711_161417, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-11 16:22:28,825][ INFO][PID:3101942] Releasing VM back to pool [2025-07-11 16:22:28,835][ INFO][PID:3101942] Retry #1 (on other host) [2025-07-11 16:22:28,837][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:28,838][ INFO][PID:3101942] VM allocation process starts [2025-07-11 16:22:28,853][ INFO][PID:3101942] Trying to allocate VM: ResallocHost, ticket_id=3896478, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-11 16:22:31,868][ INFO][PID:3101942] Allocated host ResallocHost, ticket_id=3896478, hostname=44.203.231.150, name=aws_x86_64_normalreserved_prod_05296245_20250711_161838, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-11 16:22:31,869][ INFO][PID:3101942] Allocating ssh connection to builder [2025-07-11 16:22:31,869][ INFO][PID:3101942] Checking that builder machine is OK [2025-07-11 16:22:32,283][ INFO][PID:3101942] Running remote command: copr-builder-ready srpm-builds [2025-07-11 16:22:32,489][ INFO][PID:3101942] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2025-07-11 16:22:32,489][ INFO][PID:3101942] Filling build.info file with builder info [2025-07-11 16:22:32,492][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:32,492][ INFO][PID:3101942] Not re-notifying FE and msg buses for the new host. [2025-07-11 16:22:32,493][ INFO][PID:3101942] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9267054 --detached [2025-07-11 16:22:32,842][ INFO][PID:3101942] Downloading the builder-live.log file, attempt 1 [2025-07-11 16:22:32,843][ INFO][PID:3101942] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.203.231.150 copr-rpmbuild-log [2025-07-11 16:22:37,842][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:42,843][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:47,844][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:52,846][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:22:57,847][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:23:02,848][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:23:07,849][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:23:12,850][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:23:17,851][ INFO][PID:3101942] Checking for cancel request [2025-07-11 16:23:21,460][ INFO][PID:3101942] Downloading results from builder [2025-07-11 16:23:21,461][ INFO][PID:3101942] rsyncing of mockbuilder@44.203.231.150:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054 started [2025-07-11 16:23:21,467][ INFO][PID:3101942] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.203.231.150:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/ &> /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/build-09267054.rsync.log [2025-07-11 16:23:22,050][ INFO][PID:3101942] rsyncing finished. [2025-07-11 16:23:22,052][ INFO][PID:3101942] Releasing VM back to pool [2025-07-11 16:23:22,090][ INFO][PID:3101942] Searching for 'success' file in resultdir [2025-07-11 16:23:22,090][ INFO][PID:3101942] Getting build details [2025-07-11 16:23:22,091][ INFO][PID:3101942] Retrieving SRPM info from /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054 [2025-07-11 16:23:22,092][ INFO][PID:3101942] SRPM URL: https://download.copr.fedorainfracloud.org/results/packit/containers-skopeo-2645/srpm-builds/09267054/skopeo-1.19.0-1.20250711162316995703.pr2645.58.gf0d6d0c2.src.rpm [2025-07-11 16:23:22,095][ INFO][PID:3101942] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/packit/containers-skopeo-2645/srpm-builds/09267054/skopeo-1.19.0-1.20250711162316995703.pr2645.58.gf0d6d0c2.src.rpm', 'pkg_name': 'skopeo', 'pkg_version': '102:1.19.0-1.20250711162316995703.pr2645.58.gf0d6d0c2'} [2025-07-11 16:23:22,099][ INFO][PID:3101942] Finished build: id=9267054 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/packit/containers-skopeo-2645 chroot=srpm-builds [2025-07-11 16:23:22,104][ INFO][PID:3101942] Worker succeeded build, took 49.61113238334656 [2025-07-11 16:23:22,108][ INFO][PID:3101942] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "packit", "project_name": "containers-skopeo-2645", "project_dirname": "containers-skopeo-2645", "submitter": "packit", "ended_on": 1752251002.1039817, "started_on": 1752250952.4928493, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "9267054", "build_id": 9267054, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources -p skopeo-fedora --result-dir \\\"$resultdir\\\" --pr-id 2645 --merge-pr --target-branch main --job-config-index 0 --merged-ref 'main' --no-create-symlinks https://github.com/containers/skopeo\\n\\n\", \"chroot\": \"fedora-42-x86_64\", \"builddeps\": \"packit make\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": "skopeo", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/packit/containers-skopeo-2645/srpm-builds/09267054/skopeo-1.19.0-1.20250711162316995703.pr2645.58.gf0d6d0c2.src.rpm", "uses_devel_repo": null, "sandbox": "packit/containers-skopeo-2645--packit", "results": { "name": "skopeo", "epoch": 102, "version": "1.19.0", "release": "1.20250711162316995703.pr2645.58.gf0d6d0c2", "exclusivearch": [ "x86_64", "armv3l", "armv4b", "armv4l", "armv4tl", "armv5tl", "armv5tel", "armv5tejl", "armv6l", "armv6hl", "armv7l", "armv7hl", "armv7hnl", "armv8l", "armv8hl", "armv8hnl", "armv8hcnl", "aarch64", "ppc64le", "s390x", "riscv64" ], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-42-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/packit/containers-skopeo-2645", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/containers-skopeo-2645", "result_dir": "09267054", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "102:1.19.0-1.20250711162316995703.pr2645.58.gf0d6d0c2", "id": 9267054, "mockchain_macros": { "copr_username": "packit", "copr_projectname": "containers-skopeo-2645", "vendor": "Fedora Project COPR (packit/containers-skopeo-2645)" } } ] } [2025-07-11 16:23:22,163][ INFO][PID:3101942] Sending fedora-messaging bus message in build.end [2025-07-11 16:23:22,213][ INFO][PID:3101942] Compressing /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/builder-live.log by gzip [2025-07-11 16:23:22,220][ INFO][PID:3101942] Running command 'gzip /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/builder-live.log' as PID 3150304 [2025-07-11 16:23:22,262][ INFO][PID:3101942] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/builder-live.log) [2025-07-11 16:23:22,263][ INFO][PID:3101942] Compressing /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/backend.log by gzip [2025-07-11 16:23:22,271][ INFO][PID:3101942] Running command 'gzip /var/lib/copr/public_html/results/packit/containers-skopeo-2645/srpm-builds/09267054/backend.log' as PID 3150310