[2025-06-02 21:34:52,748][ INFO][PID:2519220] Marking build as starting [2025-06-02 21:34:52,950][ INFO][PID:2519220] Trying to allocate VM: ResallocHost, ticket_id=3294176, requested_tags=['copr_builder', 'arch_x86_64'] [2025-06-02 21:38:11,720][ INFO][PID:2519220] Allocated host ResallocHost, ticket_id=3294176, hostname=3.82.194.97, name=aws_x86_64_spot_prod_04531429_20250602_213359, requested_tags=['copr_builder', 'arch_x86_64'] [2025-06-02 21:38:11,724][ INFO][PID:2519220] Allocating ssh connection to builder [2025-06-02 21:38:11,731][ INFO][PID:2519220] Checking that builder machine is OK [2025-06-02 21:38:12,072][ INFO][PID:2519220] Running remote command: copr-builder-ready centos-stream-9-x86_64 [2025-06-02 21:38:12,655][ INFO][PID:2519220] Red Hat subscription not needed for centos-stream-9-x86_64 Builder is ready to be used [2025-06-02 21:38:12,656][ INFO][PID:2519220] Filling build.info file with builder info [2025-06-02 21:38:12,666][ INFO][PID:2519220] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "packit", "project_name": "nmstate-nmstate-2730", "project_dirname": "nmstate-nmstate-2730", "submitter": "packit", "ended_on": null, "started_on": 1748900292.6663532, "submitted_on": null, "status": 3, "chroot": "centos-stream-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9119225-centos-stream-9-x86_64", "build_id": 9119225, "package_name": "nmstate", "package_version": "2.2.46-0.20250602.2527gitcafaac05", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/nmstate-nmstate-2730/nmstate", "git_hash": "4cefc2f8fbdd95406f8b9325aa96424023da1de8", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "packit/nmstate-nmstate-2730--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/packit/nmstate-nmstate-2730", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/nmstate-nmstate-2730", "result_dir": "09119225-nmstate", "built_packages": "", "id": 9119225, "mockchain_macros": { "copr_username": "packit", "copr_projectname": "nmstate-nmstate-2730", "vendor": "Fedora Project COPR (packit/nmstate-nmstate-2730)" } } ] } [2025-06-02 21:38:12,714][ INFO][PID:2519220] Sending fedora-messaging bus message in build.start [2025-06-02 21:38:13,738][ INFO][PID:2519220] Sending fedora-messaging bus message in chroot.start [2025-06-02 21:38:13,805][ INFO][PID:2519220] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9119225-centos-stream-9-x86_64 --chroot centos-stream-9-x86_64 --detached [2025-06-02 21:38:14,222][ INFO][PID:2519220] Downloading the builder-live.log file, attempt 1 [2025-06-02 21:38:14,229][ INFO][PID:2519220] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.82.194.97 copr-rpmbuild-log [2025-06-02 21:38:19,257][ INFO][PID:2519220] Periodic builder liveness probe: alive [2025-06-02 21:39:36,832][ ERROR][PID:2519220] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-06-02 21:39:46,837][WARNING][PID:2519220] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-06-02 21:39:56,856][ INFO][PID:2519220] Downloading the builder-live.log file, attempt 2 [2025-06-02 21:39:56,862][ INFO][PID:2519220] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.82.194.97 copr-rpmbuild-log [2025-06-02 21:40:57,033][ ERROR][PID:2519220] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-06-02 21:41:07,053][WARNING][PID:2519220] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-06-02 21:41:17,062][ INFO][PID:2519220] Downloading the builder-live.log file, attempt 3 [2025-06-02 21:41:17,080][ INFO][PID:2519220] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.82.194.97 copr-rpmbuild-log [2025-06-02 21:42:17,230][ ERROR][PID:2519220] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-06-02 21:42:27,240][WARNING][PID:2519220] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-06-02 21:42:37,250][ INFO][PID:2519220] Downloading the builder-live.log file, attempt 4 [2025-06-02 21:42:37,262][ INFO][PID:2519220] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.82.194.97 copr-rpmbuild-log [2025-06-02 21:43:37,356][ ERROR][PID:2519220] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-06-02 21:43:47,378][ ERROR][PID:2519220] Giving up for unstable SSH, failures: 1748900386.8377788, 1748900467.0538833, 1748900547.240102, 1748900627.3778784 [2025-06-02 21:43:47,384][ ERROR][PID:2519220] Re-try request for task on 'ResallocHost, ticket_id=3294176, hostname=3.82.194.97, name=aws_x86_64_spot_prod_04531429_20250602_213359, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-06-02 21:43:47,393][ INFO][PID:2519220] Releasing VM back to pool [2025-06-02 21:43:47,491][ INFO][PID:2519220] Retry #1 (on other host) [2025-06-02 21:43:47,600][ INFO][PID:2519220] Trying to allocate VM: ResallocHost, ticket_id=3294634, requested_tags=['copr_builder', 'arch_x86_64'] [2025-06-02 21:47:06,430][ INFO][PID:2519220] Allocated host ResallocHost, ticket_id=3294634, hostname=44.200.61.254, name=aws_x86_64_spot_prod_04531766_20250602_214336, requested_tags=['copr_builder', 'arch_x86_64'] [2025-06-02 21:47:06,435][ INFO][PID:2519220] Allocating ssh connection to builder [2025-06-02 21:47:06,439][ INFO][PID:2519220] Checking that builder machine is OK [2025-06-02 21:47:06,928][ INFO][PID:2519220] Running remote command: copr-builder-ready centos-stream-9-x86_64 [2025-06-02 21:47:07,248][ INFO][PID:2519220] Red Hat subscription not needed for centos-stream-9-x86_64 Builder is ready to be used [2025-06-02 21:47:07,252][ INFO][PID:2519220] Filling build.info file with builder info [2025-06-02 21:47:07,262][ INFO][PID:2519220] Not re-notifying FE and msg buses for the new host. [2025-06-02 21:47:07,267][ INFO][PID:2519220] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9119225-centos-stream-9-x86_64 --chroot centos-stream-9-x86_64 --detached [2025-06-02 21:47:08,040][ INFO][PID:2519220] Downloading the builder-live.log file, attempt 1 [2025-06-02 21:47:08,080][ INFO][PID:2519220] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.61.254 copr-rpmbuild-log [2025-06-02 21:48:23,216][ INFO][PID:2519220] Periodic builder liveness probe: alive [2025-06-02 21:55:07,344][ INFO][PID:2519220] Downloading results from builder [2025-06-02 21:55:07,348][ INFO][PID:2519220] rsyncing of mockbuilder@44.200.61.254:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate started [2025-06-02 21:55:07,375][ INFO][PID:2519220] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.200.61.254:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/ &> /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/build-09119225.rsync.log [2025-06-02 21:55:10,369][ INFO][PID:2519220] rsyncing finished. [2025-06-02 21:55:10,392][ INFO][PID:2519220] Releasing VM back to pool [2025-06-02 21:55:10,459][ INFO][PID:2519220] Searching for 'success' file in resultdir [2025-06-02 21:55:10,464][ INFO][PID:2519220] Going to sign pkgs from source: 9119225-centos-stream-9-x86_64 in chroot: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64 [2025-06-02 21:55:10,481][ INFO][PID:2519220] Calling '/bin/sign -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -p' (attempt #1) [2025-06-02 21:55:10,686][ INFO][PID:2519220] Calling '/bin/sign -4 -h sha256 -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-2.2.46-0.20250602.2527gitcafaac05.el9.src.rpm' (attempt #1) [2025-06-02 21:55:12,023][ INFO][PID:2519220] signed rpm: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-2.2.46-0.20250602.2527gitcafaac05.el9.src.rpm [2025-06-02 21:55:12,026][ INFO][PID:2519220] Calling '/bin/sign -4 -h sha256 -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-libs-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm' (attempt #1) [2025-06-02 21:55:13,044][ INFO][PID:2519220] signed rpm: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-libs-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm [2025-06-02 21:55:13,047][ INFO][PID:2519220] Calling '/bin/sign -4 -h sha256 -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm' (attempt #1) [2025-06-02 21:55:15,089][ INFO][PID:2519220] signed rpm: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm [2025-06-02 21:55:15,091][ INFO][PID:2519220] Calling '/bin/sign -4 -h sha256 -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-static-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm' (attempt #1) [2025-06-02 21:55:16,507][ INFO][PID:2519220] signed rpm: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-static-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm [2025-06-02 21:55:16,533][ INFO][PID:2519220] Calling '/bin/sign -4 -h sha256 -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-devel-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm' (attempt #1) [2025-06-02 21:55:17,445][ INFO][PID:2519220] signed rpm: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/nmstate-devel-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm [2025-06-02 21:55:17,447][ INFO][PID:2519220] Calling '/bin/sign -4 -h sha256 -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/python3-libnmstate-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm' (attempt #1) [2025-06-02 21:55:18,147][ INFO][PID:2519220] signed rpm: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/python3-libnmstate-2.2.46-0.20250602.2527gitcafaac05.el9.x86_64.rpm [2025-06-02 21:55:18,156][ INFO][PID:2519220] Sign done [2025-06-02 21:55:18,167][ INFO][PID:2519220] Incremental createrepo run, adding 09119225-nmstate into https://download.copr.fedorainfracloud.org/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64, (auto-create-repo=True) [2025-06-02 21:55:18,173][ INFO][PID:2519220] Running command 'copr-repo --batched /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64 --add 09119225-nmstate --no-appstream-metadata' as PID 2731849 [2025-06-02 21:55:21,482][ INFO][PID:2519220] Finished after 3 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64 --add 09119225-nmstate --no-appstream-metadata) [2025-06-02 21:55:21,485][ INFO][PID:2519220] Getting build details [2025-06-02 21:55:21,489][ INFO][PID:2519220] Listing built binary packages in /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate [2025-06-02 21:55:21,491][ INFO][PID:2519220] Built packages: ['nmstate-devel 2.2.46', 'nmstate 2.2.46', 'nmstate-static 2.2.46', 'python3-libnmstate 2.2.46', 'nmstate-libs 2.2.46'] [2025-06-02 21:55:21,493][ INFO][PID:2519220] build details: {'built_packages': 'nmstate-devel 2.2.46\nnmstate 2.2.46\nnmstate-static 2.2.46\npython3-libnmstate 2.2.46\nnmstate-libs 2.2.46'} [2025-06-02 21:55:21,495][ INFO][PID:2519220] Retrieving pubkey [2025-06-02 21:55:21,498][ INFO][PID:2519220] Calling '/bin/sign -u packit#nmstate-nmstate-2730@copr.fedorahosted.org -p' (attempt #1) [2025-06-02 21:55:21,928][ INFO][PID:2519220] Added pubkey for user packit project nmstate-nmstate-2730 into: /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/pubkey.gpg [2025-06-02 21:55:21,933][ INFO][PID:2519220] Finished build: id=9119225 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/packit/nmstate-nmstate-2730 chroot=centos-stream-9-x86_64 [2025-06-02 21:55:21,965][ INFO][PID:2519220] Worker succeeded build, took 494.7025351524353 [2025-06-02 21:55:21,968][ INFO][PID:2519220] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "packit", "project_name": "nmstate-nmstate-2730", "project_dirname": "nmstate-nmstate-2730", "submitter": "packit", "ended_on": 1748901321.965229, "started_on": 1748900827.262694, "submitted_on": null, "status": 1, "chroot": "centos-stream-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9119225-centos-stream-9-x86_64", "build_id": 9119225, "package_name": "nmstate", "package_version": "2.2.46-0.20250602.2527gitcafaac05", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/nmstate-nmstate-2730/nmstate", "git_hash": "4cefc2f8fbdd95406f8b9325aa96424023da1de8", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "packit/nmstate-nmstate-2730--packit", "results": { "packages": [ { "name": "nmstate-devel", "epoch": null, "version": "2.2.46", "release": "0.20250602.2527gitcafaac05.el9", "arch": "x86_64" }, { "name": "nmstate", "epoch": null, "version": "2.2.46", "release": "0.20250602.2527gitcafaac05.el9", "arch": "x86_64" }, { "name": "nmstate", "epoch": null, "version": "2.2.46", "release": "0.20250602.2527gitcafaac05.el9", "arch": "src" }, { "name": "nmstate-static", "epoch": null, "version": "2.2.46", "release": "0.20250602.2527gitcafaac05.el9", "arch": "x86_64" }, { "name": "python3-libnmstate", "epoch": null, "version": "2.2.46", "release": "0.20250602.2527gitcafaac05.el9", "arch": "x86_64" }, { "name": "nmstate-libs", "epoch": null, "version": "2.2.46", "release": "0.20250602.2527gitcafaac05.el9", "arch": "x86_64" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/packit/nmstate-nmstate-2730", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/nmstate-nmstate-2730", "result_dir": "09119225-nmstate", "built_packages": "nmstate-devel 2.2.46\nnmstate 2.2.46\nnmstate-static 2.2.46\npython3-libnmstate 2.2.46\nnmstate-libs 2.2.46", "id": 9119225, "mockchain_macros": { "copr_username": "packit", "copr_projectname": "nmstate-nmstate-2730", "vendor": "Fedora Project COPR (packit/nmstate-nmstate-2730)" } } ] } [2025-06-02 21:55:22,031][ INFO][PID:2519220] Sending fedora-messaging bus message in build.end [2025-06-02 21:55:22,107][ INFO][PID:2519220] Compressing /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/builder-live.log by gzip [2025-06-02 21:55:22,121][ INFO][PID:2519220] Running command 'gzip /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/builder-live.log' as PID 2732448 [2025-06-02 21:55:22,349][ INFO][PID:2519220] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/builder-live.log) [2025-06-02 21:55:22,356][ INFO][PID:2519220] Compressing /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/backend.log by gzip [2025-06-02 21:55:22,360][ INFO][PID:2519220] Running command 'gzip /var/lib/copr/public_html/results/packit/nmstate-nmstate-2730/centos-stream-9-x86_64/09119225-nmstate/backend.log' as PID 2732480