[2025-07-15 19:46:43,448][ INFO][PID:1211947] Marking build as starting [2025-07-15 19:46:43,506][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:46:43,508][ INFO][PID:1211947] VM allocation process starts [2025-07-15 19:46:43,613][ INFO][PID:1211947] Trying to allocate VM: ResallocHost, ticket_id=3942976, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-15 19:46:48,622][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:46:53,623][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:46:58,624][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:01,708][ INFO][PID:1211947] Allocated host ResallocHost, ticket_id=3942976, hostname=54.166.187.180, name=aws_x86_64_spot_prod_05433093_20250715_194512, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-15 19:47:01,709][ INFO][PID:1211947] Allocating ssh connection to builder [2025-07-15 19:47:01,710][ INFO][PID:1211947] Checking that builder machine is OK [2025-07-15 19:47:02,146][ INFO][PID:1211947] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-07-15 19:47:02,364][ INFO][PID:1211947] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-07-15 19:47:02,365][ INFO][PID:1211947] Filling build.info file with builder info [2025-07-15 19:47:02,365][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:02,366][ INFO][PID:1211947] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "sergiomb", "project_name": "vboxfor23", "project_dirname": "vboxfor23", "submitter": "sergiomb", "ended_on": null, "started_on": 1752608822.3663168, "submitted_on": null, "status": 3, "chroot": "fedora-41-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9276739-fedora-41-x86_64", "build_id": 9276739, "package_name": "VirtualBox", "package_version": "7.1.10-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/sergiomb/vboxfor23/VirtualBox", "git_hash": "b65bbf3c693d46a6bf302d06b8784594ceec12f1", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "sergiomb/vboxfor23--sergiomb", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/sergiomb/vboxfor23/fedora-41-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "bootstrap": "on", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/sergiomb/vboxfor23", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/sergiomb/vboxfor23", "result_dir": "09276739-VirtualBox", "built_packages": "", "id": 9276739, "mockchain_macros": { "copr_username": "sergiomb", "copr_projectname": "vboxfor23", "vendor": "Fedora Project COPR (sergiomb/vboxfor23)" } } ] } [2025-07-15 19:47:02,397][ INFO][PID:1211947] Sending fedora-messaging bus message in build.start [2025-07-15 19:47:03,483][ INFO][PID:1211947] Sending fedora-messaging bus message in chroot.start [2025-07-15 19:47:03,551][ INFO][PID:1211947] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9276739-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-07-15 19:47:03,929][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 1 [2025-07-15 19:47:03,932][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.166.187.180 copr-rpmbuild-log [2025-07-15 19:47:08,942][ INFO][PID:1211947] Periodic builder liveness probe: alive [2025-07-15 19:47:08,943][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:13,944][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:18,945][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:23,946][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:28,949][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:33,951][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:38,953][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:43,956][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:48,957][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:53,959][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:47:58,961][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:03,963][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:08,964][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:13,966][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:18,967][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:23,968][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:28,970][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:33,975][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:38,978][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:43,979][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:48,980][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:53,981][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:48:58,982][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:03,986][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:08,987][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:13,988][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:18,989][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:23,991][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:28,992][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:33,993][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:38,996][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:43,997][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:48,998][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:54,000][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:49:59,002][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:04,003][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:09,004][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:14,005][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:19,006][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:24,007][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:29,009][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:34,010][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:39,011][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:44,015][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:49,017][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:54,018][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:59,021][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:50:59,749][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 19:51:04,024][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:09,026][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:09,751][WARNING][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 19:51:14,027][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:19,028][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:19,751][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 2 [2025-07-15 19:51:19,771][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.166.187.180 copr-rpmbuild-log [2025-07-15 19:51:24,029][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:29,030][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:34,031][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:39,033][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:44,039][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:49,042][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:54,059][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:51:59,062][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:04,065][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:09,067][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:14,069][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:19,072][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:19,847][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 19:52:24,074][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:29,076][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:29,849][WARNING][PID:1211947] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 19:52:34,079][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:39,081][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:39,850][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 3 [2025-07-15 19:52:39,852][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.166.187.180 copr-rpmbuild-log [2025-07-15 19:52:44,082][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:49,083][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:54,084][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:52:59,086][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:04,087][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:09,088][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:14,089][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:19,092][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:24,093][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:29,096][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:34,102][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:39,104][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:39,920][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 19:53:44,107][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:49,108][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:49,938][WARNING][PID:1211947] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 19:53:54,110][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:59,128][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:53:59,939][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 4 [2025-07-15 19:53:59,941][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.166.187.180 copr-rpmbuild-log [2025-07-15 19:54:04,133][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:09,136][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:14,138][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:19,151][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:24,153][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:29,161][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:34,162][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:39,164][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:44,166][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:49,167][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:54,222][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:54:59,229][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:00,012][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 19:55:04,232][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:09,233][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:10,046][ ERROR][PID:1211947] Giving up for unstable SSH, failures: 1752609069.7510324, 1752609149.8498347, 1752609229.9382803, 1752609310.046548 [2025-07-15 19:55:10,047][ ERROR][PID:1211947] Re-try request for task on 'ResallocHost, ticket_id=3942976, hostname=54.166.187.180, name=aws_x86_64_spot_prod_05433093_20250715_194512, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-15 19:55:10,048][ INFO][PID:1211947] Releasing VM back to pool [2025-07-15 19:55:10,058][ INFO][PID:1211947] Retry #1 (on other host) [2025-07-15 19:55:10,059][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:10,060][ INFO][PID:1211947] VM allocation process starts [2025-07-15 19:55:10,073][ INFO][PID:1211947] Trying to allocate VM: ResallocHost, ticket_id=3943186, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-15 19:55:13,090][ INFO][PID:1211947] Allocated host ResallocHost, ticket_id=3943186, hostname=50.16.97.149, name=aws_x86_64_spot_prod_05433347_20250715_195101, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-15 19:55:13,091][ INFO][PID:1211947] Allocating ssh connection to builder [2025-07-15 19:55:13,091][ INFO][PID:1211947] Checking that builder machine is OK [2025-07-15 19:55:13,575][ INFO][PID:1211947] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-07-15 19:55:13,782][ INFO][PID:1211947] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-07-15 19:55:13,783][ INFO][PID:1211947] Filling build.info file with builder info [2025-07-15 19:55:13,785][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:13,786][ INFO][PID:1211947] Not re-notifying FE and msg buses for the new host. [2025-07-15 19:55:13,787][ INFO][PID:1211947] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9276739-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-07-15 19:55:14,116][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 1 [2025-07-15 19:55:14,118][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.97.149 copr-rpmbuild-log [2025-07-15 19:55:19,117][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:24,118][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:29,123][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:34,124][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:39,126][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:44,129][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:49,130][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:54,131][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:55:59,133][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:04,135][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:09,137][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:14,138][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:19,145][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:24,146][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:29,147][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:34,149][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:39,150][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:44,151][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:49,152][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:54,154][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:56:59,155][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:04,156][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:09,165][ INFO][PID:1211947] Periodic builder liveness probe: alive [2025-07-15 19:57:09,166][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:14,167][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:19,168][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:24,170][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:29,172][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:34,174][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:39,175][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:44,176][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:49,177][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:54,179][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:57:59,180][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:04,181][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:09,182][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:14,186][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:19,187][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:24,189][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:29,193][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:34,194][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:39,195][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:44,196][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:49,197][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:54,198][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:58:59,199][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:04,200][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:09,201][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:14,202][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:19,203][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:24,204][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:29,206][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:34,207][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:39,208][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:44,209][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:46,516][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 19:59:49,210][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:54,211][ INFO][PID:1211947] Checking for cancel request [2025-07-15 19:59:56,517][WARNING][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 19:59:59,214][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:04,215][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:06,518][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 2 [2025-07-15 20:00:06,524][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.97.149 copr-rpmbuild-log [2025-07-15 20:00:09,216][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:14,217][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:19,218][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:24,220][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:29,221][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:34,222][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:39,224][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:44,225][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:49,226][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:54,227][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:00:59,228][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:04,229][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:06,593][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:01:09,231][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:14,233][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:16,679][WARNING][PID:1211947] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 20:01:19,234][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:24,236][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:26,680][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 3 [2025-07-15 20:01:26,683][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.97.149 copr-rpmbuild-log [2025-07-15 20:01:29,237][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:34,239][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:39,240][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:44,242][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:49,243][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:54,245][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:01:59,246][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:04,247][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:09,251][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:14,252][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:19,254][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:24,255][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:26,752][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:02:29,256][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:34,257][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:36,762][WARNING][PID:1211947] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 20:02:39,258][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:44,260][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:46,765][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 4 [2025-07-15 20:02:46,766][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@50.16.97.149 copr-rpmbuild-log [2025-07-15 20:02:49,262][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:54,263][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:02:59,265][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:04,266][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:09,270][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:14,271][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:19,274][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:24,275][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:29,276][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:34,278][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:39,279][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:44,281][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:46,850][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:03:49,282][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:54,283][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:56,855][ ERROR][PID:1211947] Giving up for unstable SSH, failures: 1752609596.5175197, 1752609676.679564, 1752609756.762935, 1752609836.8551178 [2025-07-15 20:03:56,858][ ERROR][PID:1211947] Re-try request for task on 'ResallocHost, ticket_id=3943186, hostname=50.16.97.149, name=aws_x86_64_spot_prod_05433347_20250715_195101, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-15 20:03:56,861][ INFO][PID:1211947] Releasing VM back to pool [2025-07-15 20:03:56,889][ INFO][PID:1211947] Retry #2 (on other host) [2025-07-15 20:03:56,891][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:03:56,896][ INFO][PID:1211947] VM allocation process starts [2025-07-15 20:03:56,936][ INFO][PID:1211947] Trying to allocate VM: ResallocHost, ticket_id=3943398, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-15 20:03:59,997][ INFO][PID:1211947] Allocated host ResallocHost, ticket_id=3943398, hostname=54.209.245.127, name=aws_x86_64_spot_prod_05433898_20250715_200227, requested_tags=['copr_builder', 'arch_x86_64'] [2025-07-15 20:03:59,998][ INFO][PID:1211947] Allocating ssh connection to builder [2025-07-15 20:03:59,999][ INFO][PID:1211947] Checking that builder machine is OK [2025-07-15 20:04:00,476][ INFO][PID:1211947] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-07-15 20:04:00,686][ INFO][PID:1211947] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-07-15 20:04:00,687][ INFO][PID:1211947] Filling build.info file with builder info [2025-07-15 20:04:00,716][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:00,717][ INFO][PID:1211947] Not re-notifying FE and msg buses for the new host. [2025-07-15 20:04:00,718][ INFO][PID:1211947] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9276739-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-07-15 20:04:01,065][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 1 [2025-07-15 20:04:01,068][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.127 copr-rpmbuild-log [2025-07-15 20:04:06,066][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:11,067][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:16,069][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:21,073][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:26,074][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:31,075][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:36,076][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:41,077][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:46,079][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:51,084][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:04:56,086][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:01,090][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:06,092][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:11,093][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:16,094][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:21,095][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:26,096][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:31,100][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:36,102][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:41,103][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:46,104][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:51,107][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:05:56,108][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:01,117][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:06,118][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:11,120][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:16,124][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:21,126][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:26,129][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:31,130][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:36,133][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:41,134][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:46,136][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:51,139][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:06:56,140][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:01,141][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:06,144][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:11,156][ INFO][PID:1211947] Periodic builder liveness probe: alive [2025-07-15 20:07:11,156][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:16,157][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:21,158][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:26,160][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:31,162][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:36,166][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:41,170][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:46,171][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:51,172][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:07:56,174][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:01,179][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:06,185][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:11,189][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:16,190][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:21,192][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:26,195][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:31,206][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:36,207][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:41,212][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:46,218][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:51,220][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:08:56,221][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:01,227][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:06,231][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:11,238][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:16,241][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:21,253][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:26,256][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:31,263][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:36,265][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:41,273][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:46,279][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:51,284][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:09:56,291][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:01,296][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:06,307][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:11,309][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:16,322][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:21,323][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:26,325][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:31,331][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:36,333][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:41,334][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:46,343][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:51,350][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:10:56,352][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:01,356][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:06,361][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:11,373][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:16,376][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:21,377][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:26,386][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:31,391][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:36,396][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:41,406][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:46,412][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:51,416][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:11:56,421][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:01,430][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:06,437][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:11,451][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:16,453][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:21,460][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:26,463][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:31,475][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:36,477][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:41,479][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:46,492][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:51,500][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:12:56,502][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:01,510][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:06,512][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:11,513][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:16,518][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:21,525][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:26,526][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:31,532][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:36,534][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:41,547][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:46,553][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:51,556][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:13:56,563][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:01,576][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:06,578][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:11,592][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:16,594][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:21,603][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:26,606][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:31,608][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:36,610][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:41,613][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:46,625][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:51,652][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:14:56,660][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:01,661][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:06,664][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:11,679][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:16,683][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:21,687][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:26,690][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:31,692][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:36,701][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:41,706][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:46,708][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:51,713][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:15:56,715][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:01,729][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:06,737][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:11,741][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:16,760][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:21,761][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:26,771][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:31,780][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:36,785][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:41,790][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:46,800][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:51,802][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:16:56,808][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:01,838][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:06,845][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:11,878][ INFO][PID:1211947] Periodic builder liveness probe: alive [2025-07-15 20:17:11,879][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:16,895][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:21,896][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:26,902][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:31,904][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:36,917][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:41,922][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:46,925][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:51,945][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:17:56,958][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:01,964][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:06,974][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:11,979][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:16,982][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:21,990][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:27,012][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:32,017][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:37,037][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:42,039][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:47,045][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:52,047][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:18:57,063][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:02,069][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:07,072][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:12,077][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:17,088][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:22,091][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:27,099][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:32,110][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:37,116][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:42,121][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:47,125][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:52,127][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:19:57,129][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:02,137][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:07,143][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:12,145][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:17,147][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:22,154][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:27,157][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:32,167][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:37,189][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:42,198][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:47,201][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:52,204][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:20:57,206][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:02,216][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:07,222][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:12,225][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:17,231][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:22,238][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:27,251][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:32,252][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:37,254][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:42,257][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:47,266][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:52,268][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:21:57,270][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:02,271][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:07,277][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:12,283][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:17,290][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:22,291][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:27,292][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:32,301][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:37,302][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:42,303][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:47,306][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:52,312][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:22:57,313][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:02,316][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:07,317][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:12,319][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:17,323][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:22,324][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:27,325][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:32,327][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:37,328][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:42,332][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:47,333][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:52,334][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:23:57,339][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:02,343][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:07,344][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:12,346][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:17,348][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:22,352][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:27,356][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:32,358][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:37,368][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:42,376][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:47,380][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:52,383][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:24:57,387][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:02,389][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:07,394][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:12,396][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:17,398][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:22,399][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:27,401][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:32,402][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:37,403][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:42,406][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:47,411][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:52,416][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:25:57,417][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:02,418][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:07,419][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:12,420][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:17,421][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:22,422][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:27,424][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:32,425][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:37,426][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:42,427][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:47,428][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:52,430][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:26:57,431][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:02,432][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:07,433][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:12,441][ INFO][PID:1211947] Periodic builder liveness probe: alive [2025-07-15 20:27:12,441][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:17,443][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:22,444][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:27,445][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:32,447][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:37,448][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:42,449][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:47,451][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:52,454][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:27:57,463][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:02,465][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:07,466][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:12,467][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:17,468][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:22,469][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:27,470][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:32,471][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:37,474][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:42,475][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:47,477][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:52,478][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:28:57,481][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:02,482][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:07,483][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:12,485][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:17,487][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:22,489][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:27,490][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:32,493][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:37,494][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:42,495][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:47,497][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:52,498][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:29:57,500][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:02,503][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:07,508][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:12,509][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:17,510][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:22,511][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:27,512][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:32,514][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:37,515][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:42,516][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:47,517][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:52,519][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:30:57,520][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:02,521][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:07,522][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:12,529][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:17,534][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:22,535][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:27,536][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:32,537][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:37,539][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:42,541][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:47,542][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:52,543][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:31:57,545][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:02,546][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:07,547][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:12,548][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:17,572][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:22,573][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:27,574][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:32,576][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:37,578][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:42,579][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:47,582][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:52,583][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:32:57,584][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:02,585][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:07,587][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:12,590][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:17,596][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:22,598][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:27,600][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:32,601][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:37,602][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:42,603][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:47,604][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:52,605][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:33:57,607][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:02,608][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:07,609][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:12,611][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:17,088][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:34:17,612][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:22,616][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:27,104][WARNING][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 20:34:27,626][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:32,628][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:37,138][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 2 [2025-07-15 20:34:37,404][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.127 copr-rpmbuild-log [2025-07-15 20:34:37,407][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:34:37,630][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:42,632][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:47,412][WARNING][PID:1211947] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 20:34:47,633][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:52,634][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:34:57,413][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 3 [2025-07-15 20:34:57,416][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.127 copr-rpmbuild-log [2025-07-15 20:34:57,428][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:34:57,635][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:02,638][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:07,462][WARNING][PID:1211947] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-15 20:35:07,639][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:12,640][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:17,463][ INFO][PID:1211947] Downloading the builder-live.log file, attempt 4 [2025-07-15 20:35:17,465][ INFO][PID:1211947] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.209.245.127 copr-rpmbuild-log [2025-07-15 20:35:17,642][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:22,643][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:27,645][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:32,647][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:37,648][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:42,650][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:47,651][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:52,652][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:35:57,653][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:36:02,654][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:36:07,655][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:36:12,656][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:36:17,533][ ERROR][PID:1211947] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-15 20:36:17,657][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:36:22,658][ INFO][PID:1211947] Checking for cancel request [2025-07-15 20:36:27,643][ ERROR][PID:1211947] Giving up for unstable SSH, failures: 1752611667.1019142, 1752611687.4123354, 1752611707.4625893, 1752611787.6222806 [2025-07-15 20:36:27,703][ ERROR][PID:1211947] Re-try request for task on 'ResallocHost, ticket_id=3943398, hostname=54.209.245.127, name=aws_x86_64_spot_prod_05433898_20250715_200227, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-15 20:36:27,706][ INFO][PID:1211947] Releasing VM back to pool [2025-07-15 20:36:27,753][ ERROR][PID:1211947] Backend process error: Three host tried without success: {'54.209.245.127', '50.16.97.149', '54.166.187.180'} [2025-07-15 20:36:27,783][WARNING][PID:1211947] Switching not-finished job state to 'failed' [2025-07-15 20:36:27,784][ INFO][PID:1211947] Worker failed build, took 1947.0426180362701 [2025-07-15 20:36:27,813][ INFO][PID:1211947] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "sergiomb", "project_name": "vboxfor23", "project_dirname": "vboxfor23", "submitter": "sergiomb", "ended_on": 1752611787.7600589, "started_on": 1752609840.7174408, "submitted_on": null, "status": 0, "chroot": "fedora-41-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9276739-fedora-41-x86_64", "build_id": 9276739, "package_name": "VirtualBox", "package_version": "7.1.10-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/sergiomb/vboxfor23/VirtualBox", "git_hash": "b65bbf3c693d46a6bf302d06b8784594ceec12f1", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "sergiomb/vboxfor23--sergiomb", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/sergiomb/vboxfor23/fedora-41-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "bootstrap": "on", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/sergiomb/vboxfor23", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/sergiomb/vboxfor23", "result_dir": "09276739-VirtualBox", "built_packages": "", "id": 9276739, "mockchain_macros": { "copr_username": "sergiomb", "copr_projectname": "vboxfor23", "vendor": "Fedora Project COPR (sergiomb/vboxfor23)" } } ] } [2025-07-15 20:36:28,796][ INFO][PID:1211947] Sending fedora-messaging bus message in build.end [2025-07-15 20:36:29,146][ INFO][PID:1211947] Compressing /var/lib/copr/public_html/results/sergiomb/vboxfor23/fedora-41-x86_64/09276739-VirtualBox/builder-live.log by gzip [2025-07-15 20:36:29,157][ INFO][PID:1211947] Running command 'gzip /var/lib/copr/public_html/results/sergiomb/vboxfor23/fedora-41-x86_64/09276739-VirtualBox/builder-live.log' as PID 1603145 [2025-07-15 20:36:29,173][ INFO][PID:1211947] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/sergiomb/vboxfor23/fedora-41-x86_64/09276739-VirtualBox/builder-live.log) [2025-07-15 20:36:29,180][ INFO][PID:1211947] Compressing /var/lib/copr/public_html/results/sergiomb/vboxfor23/fedora-41-x86_64/09276739-VirtualBox/backend.log by gzip [2025-07-15 20:36:29,182][ INFO][PID:1211947] Running command 'gzip /var/lib/copr/public_html/results/sergiomb/vboxfor23/fedora-41-x86_64/09276739-VirtualBox/backend.log' as PID 1603149