[2025-05-19 14:33:54,831][ INFO][PID:3529336] Marking build as starting [2025-05-19 14:33:54,891][ INFO][PID:3529336] Trying to allocate VM: ResallocHost, ticket_id=3079496, requested_tags=['copr_builder', 'arch_x86_64'] [2025-05-19 14:33:57,926][ INFO][PID:3529336] Allocated host ResallocHost, ticket_id=3079496, hostname=18.206.117.139, name=aws_x86_64_spot_prod_04154064_20250519_142118, requested_tags=['copr_builder', 'arch_x86_64'] [2025-05-19 14:33:57,927][ INFO][PID:3529336] Allocating ssh connection to builder [2025-05-19 14:33:57,929][ INFO][PID:3529336] Checking that builder machine is OK [2025-05-19 14:33:58,333][ INFO][PID:3529336] Running remote command: copr-builder-ready epel-9-x86_64 [2025-05-19 14:33:58,567][ INFO][PID:3529336] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-05-19 14:33:58,571][ INFO][PID:3529336] Filling build.info file with builder info [2025-05-19 14:33:58,575][ INFO][PID:3529336] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": null, "started_on": 1747665238.5750623, "submitted_on": null, "status": 3, "chroot": "epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9063035-epel-9-x86_64", "build_id": 9063035, "package_name": "python-tencentcloud-sdk-python-cwp", "package_version": "3.0.1382-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-tencentcloud-sdk-python-cwp", "git_hash": "78ea4539c0a24607eca17f958ad07f26327e71b1", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "09063035-python-tencentcloud-sdk-python-cwp", "built_packages": "", "id": 9063035, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2025-05-19 14:33:58,620][ INFO][PID:3529336] Sending fedora-messaging bus message in build.start [2025-05-19 14:34:00,085][ INFO][PID:3529336] Sending fedora-messaging bus message in chroot.start [2025-05-19 14:34:00,137][ INFO][PID:3529336] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9063035-epel-9-x86_64 --chroot epel-9-x86_64 --detached [2025-05-19 14:34:00,546][ INFO][PID:3529336] Downloading the builder-live.log file, attempt 1 [2025-05-19 14:34:00,550][ INFO][PID:3529336] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.206.117.139 copr-rpmbuild-log [2025-05-19 14:34:05,562][ INFO][PID:3529336] Periodic builder liveness probe: alive [2025-05-19 14:34:06,836][ ERROR][PID:3529336] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-05-19 14:34:16,838][WARNING][PID:3529336] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-05-19 14:34:26,840][ INFO][PID:3529336] Downloading the builder-live.log file, attempt 2 [2025-05-19 14:34:26,842][ INFO][PID:3529336] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.206.117.139 copr-rpmbuild-log [2025-05-19 14:35:26,911][ ERROR][PID:3529336] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-05-19 14:35:36,913][WARNING][PID:3529336] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-05-19 14:35:46,916][ INFO][PID:3529336] Downloading the builder-live.log file, attempt 3 [2025-05-19 14:35:46,922][ INFO][PID:3529336] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.206.117.139 copr-rpmbuild-log [2025-05-19 14:36:47,026][ ERROR][PID:3529336] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-05-19 14:36:57,048][WARNING][PID:3529336] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-05-19 14:37:07,051][ INFO][PID:3529336] Downloading the builder-live.log file, attempt 4 [2025-05-19 14:37:07,061][ INFO][PID:3529336] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.206.117.139 copr-rpmbuild-log [2025-05-19 14:38:07,145][ ERROR][PID:3529336] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-05-19 14:38:17,150][ ERROR][PID:3529336] Giving up for unstable SSH, failures: 1747665256.8385313, 1747665336.9136167, 1747665417.048579, 1747665497.1505313 [2025-05-19 14:38:17,159][ ERROR][PID:3529336] Re-try request for task on 'ResallocHost, ticket_id=3079496, hostname=18.206.117.139, name=aws_x86_64_spot_prod_04154064_20250519_142118, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-05-19 14:38:17,160][ INFO][PID:3529336] Releasing VM back to pool [2025-05-19 14:38:17,223][ INFO][PID:3529336] Retry #1 (on other host) [2025-05-19 14:38:17,308][ INFO][PID:3529336] Trying to allocate VM: ResallocHost, ticket_id=3079659, requested_tags=['copr_builder', 'arch_x86_64'] [2025-05-19 14:39:05,704][ INFO][PID:3529336] Allocated host ResallocHost, ticket_id=3079659, hostname=2620:52:3:1:dead:beef:cafe:c18f, name=copr_hv_x86_64_03_prod_04154552_20250519_142755, requested_tags=['copr_builder', 'arch_x86_64'] [2025-05-19 14:39:05,708][ INFO][PID:3529336] Allocating ssh connection to builder [2025-05-19 14:39:05,710][ INFO][PID:3529336] Checking that builder machine is OK [2025-05-19 14:39:06,658][ INFO][PID:3529336] Running remote command: copr-builder-ready epel-9-x86_64 [2025-05-19 14:39:07,232][ INFO][PID:3529336] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-05-19 14:39:07,234][ INFO][PID:3529336] Filling build.info file with builder info [2025-05-19 14:39:07,241][ INFO][PID:3529336] Not re-notifying FE and msg buses for the new host. [2025-05-19 14:39:07,242][ INFO][PID:3529336] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9063035-epel-9-x86_64 --chroot epel-9-x86_64 --detached [2025-05-19 14:39:08,013][ INFO][PID:3529336] Downloading the builder-live.log file, attempt 1 [2025-05-19 14:39:08,020][ INFO][PID:3529336] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c18f copr-rpmbuild-log [2025-05-19 14:41:31,737][ INFO][PID:3529336] Downloading results from builder [2025-05-19 14:41:31,740][ INFO][PID:3529336] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c18f]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp started [2025-05-19 14:41:31,743][ INFO][PID:3529336] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c18f]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp/ &> /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp/build-09063035.rsync.log [2025-05-19 14:41:32,318][ INFO][PID:3529336] rsyncing finished. [2025-05-19 14:41:32,328][ INFO][PID:3529336] Releasing VM back to pool [2025-05-19 14:41:32,356][ INFO][PID:3529336] Searching for 'success' file in resultdir [2025-05-19 14:41:32,359][ ERROR][PID:3529336] Build failed: Backend process error: No success file => build failure [2025-05-19 14:41:32,364][ INFO][PID:3529336] Finished build: id=9063035 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/PyPI chroot=epel-9-x86_64 [2025-05-19 14:41:32,372][ ERROR][PID:3529336] Backend process error: No success file => build failure [2025-05-19 14:41:32,374][ INFO][PID:3529336] Worker failed build, took 145.1338620185852 [2025-05-19 14:41:32,376][ INFO][PID:3529336] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@copr", "project_name": "PyPI", "project_dirname": "PyPI", "submitter": "ksurma", "ended_on": 1747665692.3748438, "started_on": 1747665547.2409818, "submitted_on": null, "status": 0, "chroot": "epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9063035-epel-9-x86_64", "build_id": 9063035, "package_name": "python-tencentcloud-sdk-python-cwp", "package_version": "3.0.1382-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/PyPI/python-tencentcloud-sdk-python-cwp", "git_hash": "78ea4539c0a24607eca17f958ad07f26327e71b1", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@copr/PyPI--ksurma", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI/epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@copr/PyPI", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/PyPI", "result_dir": "09063035-python-tencentcloud-sdk-python-cwp", "built_packages": "", "id": 9063035, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "PyPI", "vendor": "Fedora Project COPR (@copr/PyPI)" } } ] } [2025-05-19 14:41:32,414][ INFO][PID:3529336] Sending fedora-messaging bus message in build.end [2025-05-19 14:41:32,541][ INFO][PID:3529336] Compressing /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp/builder-live.log by gzip [2025-05-19 14:41:32,545][ INFO][PID:3529336] Running command 'gzip /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp/builder-live.log' as PID 3581014 [2025-05-19 14:41:32,551][ INFO][PID:3529336] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp/builder-live.log) [2025-05-19 14:41:32,557][ INFO][PID:3529336] Compressing /var/lib/copr/public_html/results/@copr/PyPI/epel-9-x86_64/09063035-python-tencentcloud-sdk-python-cwp/backend.log by gzip