[2025-07-15 20:43:24,987][  INFO][PID:1636457] Marking build as starting
[2025-07-15 20:43:25,013][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:25,013][  INFO][PID:1636457] VM allocation process starts
[2025-07-15 20:43:25,086][  INFO][PID:1636457] Trying to allocate VM: ResallocHost, ticket_id=3945093, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-15 20:43:28,144][  INFO][PID:1636457] Allocated host ResallocHost, ticket_id=3945093, hostname=3.236.16.145, name=aws_x86_64_spot_prod_05434711_20250715_203026, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-15 20:43:28,144][  INFO][PID:1636457] Allocating ssh connection to builder
[2025-07-15 20:43:28,144][  INFO][PID:1636457] Checking that builder machine is OK
[2025-07-15 20:43:28,630][  INFO][PID:1636457] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-07-15 20:43:28,829][  INFO][PID:1636457] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-07-15 20:43:28,829][  INFO][PID:1636457] Filling build.info file with builder info
[2025-07-15 20:43:28,830][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:28,831][  INFO][PID:1636457] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "lecris",
            "project_name": "cmake-ninja",
            "project_dirname": "cmake-ninja",
            "submitter": "lecris",
            "ended_on": null,
            "started_on": 1752612208.831399,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9276940-fedora-rawhide-x86_64",
            "build_id": 9276940,
            "package_name": "mathgl",
            "package_version": "8.0.1-21",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lecris/cmake-ninja/mathgl",
            "git_hash": "1ebb96f32371fe11bd3c51e7b96ea67b1f1619c2",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "lecris/cmake-ninja--lecris",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-ninja/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/lecris/cmake-ninja",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-ninja",
            "result_dir": "09276940-mathgl",
            "built_packages": "",
            "id": 9276940,
            "mockchain_macros": {
                "copr_username": "lecris",
                "copr_projectname": "cmake-ninja",
                "vendor": "Fedora Project COPR (lecris/cmake-ninja)"
            }
        }
    ]
}
[2025-07-15 20:43:28,860][  INFO][PID:1636457] Sending fedora-messaging bus message in build.start
[2025-07-15 20:43:29,618][  INFO][PID:1636457] Sending fedora-messaging bus message in chroot.start
[2025-07-15 20:43:29,644][  INFO][PID:1636457] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9276940-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-07-15 20:43:29,972][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 1
[2025-07-15 20:43:29,974][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.16.145 copr-rpmbuild-log
[2025-07-15 20:43:34,981][  INFO][PID:1636457] Periodic builder liveness probe: alive
[2025-07-15 20:43:34,982][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:39,983][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:44,985][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:49,986][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:54,987][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:43:59,988][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:04,989][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:09,991][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:14,992][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:19,993][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:24,994][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:29,995][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:34,996][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:39,997][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:44,998][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:49,999][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:44:55,000][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:00,002][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:05,003][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:10,004][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:15,005][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:20,007][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:25,008][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:30,009][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:35,010][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:40,011][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:45,013][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:50,014][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:45:55,016][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:00,020][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:05,021][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:10,022][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:15,023][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:20,024][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:25,026][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:30,027][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:35,028][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:40,029][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:45,030][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:50,031][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:46:55,032][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:00,033][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:05,035][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:10,036][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:15,037][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:20,039][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:25,040][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:30,041][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:35,042][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:40,043][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:45,044][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:50,045][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:47:55,047][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:00,048][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:05,049][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:10,051][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:15,053][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:20,054][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:25,055][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:30,056][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:35,057][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:40,058][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:45,059][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:50,060][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:48:55,061][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:00,062][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:05,063][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:10,064][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:15,069][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:20,070][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:25,071][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:30,072][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:35,074][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:40,075][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:45,076][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:50,077][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:49:55,078][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:00,079][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:05,080][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:10,081][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:15,083][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:20,085][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:25,086][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:30,087][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:35,089][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:40,090][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:45,091][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:50,092][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:50:55,094][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:00,095][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:05,096][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:10,097][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:15,098][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:20,103][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:25,104][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:30,106][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:35,107][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:40,108][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:45,109][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:50,111][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:51:55,116][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:00,118][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:05,119][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:10,120][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:15,121][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:20,123][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:25,124][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:30,126][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:35,127][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:40,128][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:45,129][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:50,130][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:52:55,131][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:00,133][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:05,134][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:10,135][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:15,137][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:20,138][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:23,551][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 20:53:25,140][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:30,141][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:33,552][WARNING][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 20:53:35,150][  INFO][PID:1636457] Periodic builder liveness probe: alive
[2025-07-15 20:53:35,150][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:40,151][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:43,552][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 2
[2025-07-15 20:53:43,555][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.16.145 copr-rpmbuild-log
[2025-07-15 20:53:45,152][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:50,153][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:53:55,154][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:00,155][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:05,156][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:10,157][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:15,159][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:20,161][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:25,162][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:30,163][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:35,164][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:40,166][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:43,623][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 20:54:45,167][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:50,168][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:54:53,730][WARNING][PID:1636457] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 20:54:55,169][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:00,170][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:03,731][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 3
[2025-07-15 20:55:03,732][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.16.145 copr-rpmbuild-log
[2025-07-15 20:55:05,171][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:10,172][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:15,173][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:20,174][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:25,175][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:30,177][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:35,178][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:40,179][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:45,180][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:50,181][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:55:55,182][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:00,183][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:03,803][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 20:56:05,185][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:10,186][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:13,806][WARNING][PID:1636457] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 20:56:15,187][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:20,188][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:23,806][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 4
[2025-07-15 20:56:23,807][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.16.145 copr-rpmbuild-log
[2025-07-15 20:56:25,189][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:30,190][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:35,191][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:40,192][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:45,193][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:50,194][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:56:55,195][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:00,196][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:05,197][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:10,198][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:15,200][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:20,200][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:23,875][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 20:57:25,201][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:30,202][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:33,877][ ERROR][PID:1636457] Giving up for unstable SSH, failures: 1752612813.5521073, 1752612893.7305264, 1752612973.8059576, 1752613053.8775566
[2025-07-15 20:57:33,878][ ERROR][PID:1636457] Re-try request for task on 'ResallocHost, ticket_id=3945093, hostname=3.236.16.145, name=aws_x86_64_spot_prod_05434711_20250715_203026, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-15 20:57:33,879][  INFO][PID:1636457] Releasing VM back to pool
[2025-07-15 20:57:33,892][  INFO][PID:1636457] Retry #1 (on other host)
[2025-07-15 20:57:33,893][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:33,893][  INFO][PID:1636457] VM allocation process starts
[2025-07-15 20:57:33,929][  INFO][PID:1636457] Trying to allocate VM: ResallocHost, ticket_id=3945426, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-15 20:57:36,940][  INFO][PID:1636457] Allocated host ResallocHost, ticket_id=3945426, hostname=54.237.235.248, name=aws_x86_64_spot_prod_05435272_20250715_205320, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-15 20:57:36,940][  INFO][PID:1636457] Allocating ssh connection to builder
[2025-07-15 20:57:36,941][  INFO][PID:1636457] Checking that builder machine is OK
[2025-07-15 20:57:37,392][  INFO][PID:1636457] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-07-15 20:57:37,587][  INFO][PID:1636457] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-07-15 20:57:37,587][  INFO][PID:1636457] Filling build.info file with builder info
[2025-07-15 20:57:37,606][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:37,607][  INFO][PID:1636457] Not re-notifying FE and msg buses for the new host.
[2025-07-15 20:57:37,608][  INFO][PID:1636457] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9276940-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-07-15 20:57:37,927][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 1
[2025-07-15 20:57:37,928][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.237.235.248 copr-rpmbuild-log
[2025-07-15 20:57:42,928][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:47,929][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:52,930][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:57:57,932][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:02,933][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:07,934][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:12,935][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:17,936][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:22,937][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:27,938][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:32,939][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:37,940][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:42,941][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:47,942][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:52,943][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:58:57,944][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:02,946][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:07,948][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:12,949][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:17,950][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:22,951][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:27,956][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:32,957][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:37,959][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:42,961][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:47,962][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:52,963][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 20:59:57,965][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:02,973][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:07,975][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:12,976][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:17,977][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:22,979][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:27,980][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:32,981][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:37,982][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:42,984][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:47,985][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:52,986][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:00:57,988][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:02,989][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:07,990][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:12,992][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:17,993][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:22,995][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:27,998][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:33,003][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:38,004][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:43,006][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:48,007][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:53,012][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:01:58,016][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:03,017][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:08,018][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:13,020][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:18,021][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:23,023][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:28,028][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:33,029][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:38,030][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:43,031][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:48,033][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:53,034][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:02:58,036][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:03,037][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:08,041][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:13,042][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:18,043][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:23,045][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:28,046][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:33,047][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:38,057][  INFO][PID:1636457] Periodic builder liveness probe: alive
[2025-07-15 21:03:38,058][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:43,059][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:48,060][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:53,061][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:03:58,062][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:03,063][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:08,064][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:13,066][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:18,068][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:23,069][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:28,070][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:33,071][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:38,072][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:43,074][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:48,076][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:53,081][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:04:58,083][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:03,084][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:08,085][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:13,087][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:18,088][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:23,089][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:28,090][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:33,100][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:38,102][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:43,105][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:48,109][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:53,111][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:05:58,112][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:03,113][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:08,114][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:13,116][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:18,117][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:23,119][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:28,122][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:33,124][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:38,126][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:43,136][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:48,137][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:53,139][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:06:58,140][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:03,141][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:08,142][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:13,144][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:18,146][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:23,148][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:28,149][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:33,150][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:38,152][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:43,153][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:48,155][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:53,156][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:07:58,157][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:03,162][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:08,163][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:13,165][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:18,167][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:23,169][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:28,170][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:33,179][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:38,180][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:43,181][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:48,184][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:53,186][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:08:58,188][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:03,189][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:08,190][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:13,199][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:18,202][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:23,204][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:28,205][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:33,207][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:38,210][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:43,211][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:47,493][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 21:09:48,213][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:53,214][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:09:57,500][WARNING][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 21:09:58,216][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:03,217][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:07,501][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 2
[2025-07-15 21:10:07,503][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.237.235.248 copr-rpmbuild-log
[2025-07-15 21:10:08,218][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:13,219][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:18,220][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:23,223][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:28,226][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:33,227][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:38,228][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:43,231][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:48,232][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:53,234][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:10:58,237][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:03,238][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:07,573][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 21:11:08,240][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:13,243][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:17,588][WARNING][PID:1636457] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 21:11:18,245][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:23,247][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:27,600][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 3
[2025-07-15 21:11:27,606][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.237.235.248 copr-rpmbuild-log
[2025-07-15 21:11:28,251][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:33,261][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:38,262][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:43,271][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:48,279][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:53,282][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:11:58,284][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:03,290][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:08,294][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:13,301][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:18,304][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:23,308][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:27,724][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 21:12:28,309][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:33,313][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:37,733][WARNING][PID:1636457] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-15 21:12:38,317][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:43,320][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:47,734][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 4
[2025-07-15 21:12:47,736][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.237.235.248 copr-rpmbuild-log
[2025-07-15 21:12:48,328][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:53,330][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:12:58,338][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:03,348][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:08,352][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:13,354][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:18,358][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:23,363][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:28,370][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:33,373][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:38,411][  INFO][PID:1636457] Periodic builder liveness probe: alive
[2025-07-15 21:13:38,412][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:43,413][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:47,812][ ERROR][PID:1636457] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-15 21:13:48,415][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:53,417][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:57,817][ ERROR][PID:1636457] Giving up for unstable SSH, failures: 1752613797.5004213, 1752613877.5888298, 1752613957.7338684, 1752614037.8169787
[2025-07-15 21:13:57,820][ ERROR][PID:1636457] Re-try request for task on 'ResallocHost, ticket_id=3945426, hostname=54.237.235.248, name=aws_x86_64_spot_prod_05435272_20250715_205320, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-15 21:13:57,824][  INFO][PID:1636457] Releasing VM back to pool
[2025-07-15 21:13:57,850][  INFO][PID:1636457] Retry #2 (on other host)
[2025-07-15 21:13:57,851][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:13:57,853][  INFO][PID:1636457] VM allocation process starts
[2025-07-15 21:13:57,881][  INFO][PID:1636457] Trying to allocate VM: ResallocHost, ticket_id=3945956, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-15 21:14:02,884][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:07,889][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:12,895][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:17,898][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:22,904][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:27,907][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:32,908][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:37,910][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:42,912][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:47,915][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:52,919][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:14:57,920][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:02,925][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:07,930][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:12,933][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:17,934][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:22,936][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:26,096][  INFO][PID:1636457] Allocated host ResallocHost, ticket_id=3945956, hostname=44.202.70.43, name=aws_x86_64_spot_prod_05435717_20250715_211105, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-15 21:15:26,097][  INFO][PID:1636457] Allocating ssh connection to builder
[2025-07-15 21:15:26,101][  INFO][PID:1636457] Checking that builder machine is OK
[2025-07-15 21:15:26,352][  INFO][PID:1636457] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-07-15 21:15:26,579][  INFO][PID:1636457] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-07-15 21:15:26,581][  INFO][PID:1636457] Filling build.info file with builder info
[2025-07-15 21:15:26,587][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:26,588][  INFO][PID:1636457] Not re-notifying FE and msg buses for the new host.
[2025-07-15 21:15:26,589][  INFO][PID:1636457] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9276940-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-07-15 21:15:26,925][  INFO][PID:1636457] Downloading the builder-live.log file, attempt 1
[2025-07-15 21:15:26,926][  INFO][PID:1636457] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.202.70.43 copr-rpmbuild-log
[2025-07-15 21:15:31,927][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:36,929][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:41,930][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:46,931][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:51,934][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:15:56,936][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:01,937][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:06,939][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:11,941][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:16,943][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:21,944][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:26,945][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:31,946][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:36,948][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:41,950][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:46,952][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:51,954][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:16:56,960][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:01,962][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:06,964][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:11,965][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:16,967][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:21,968][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:26,969][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:31,970][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:36,974][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:41,975][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:46,977][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:51,979][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:17:56,980][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:01,981][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:06,983][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:11,984][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:16,985][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:21,986][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:26,987][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:31,989][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:36,990][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:41,991][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:46,993][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:51,994][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:18:56,995][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:01,996][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:06,997][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:11,998][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:16,999][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:22,000][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:27,002][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:32,004][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:37,005][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:42,006][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:47,009][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:52,010][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:19:57,013][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:02,014][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:07,015][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:12,016][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:17,017][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:22,018][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:27,019][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:32,021][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:37,022][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:42,023][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:47,024][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:52,025][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:20:57,027][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:02,028][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:07,029][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:12,030][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:17,031][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:22,032][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:27,034][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:32,035][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:37,036][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:42,037][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:47,038][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:52,039][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:21:57,040][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:02,041][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:07,042][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:12,043][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:17,044][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:22,045][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:27,047][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:32,048][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:37,049][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:42,050][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:47,051][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:52,052][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:22:57,054][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:02,055][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:07,056][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:12,057][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:17,058][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:22,059][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:27,061][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:32,062][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:37,063][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:42,069][  INFO][PID:1636457] Periodic builder liveness probe: alive
[2025-07-15 21:23:42,069][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:47,079][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:52,086][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:23:57,087][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:02,088][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:07,091][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:12,092][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:17,093][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:22,094][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:27,096][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:32,097][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:37,099][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:42,100][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:47,101][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:52,108][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:24:57,115][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:02,116][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:07,117][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:12,118][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:17,120][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:22,121][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:27,123][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:32,127][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:37,129][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:42,132][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:47,134][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:52,138][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:25:57,140][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:02,142][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:07,146][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:12,147][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:17,149][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:22,150][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:27,151][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:32,152][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:37,153][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:42,155][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:47,156][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:52,157][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:26:57,159][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:02,160][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:07,161][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:12,162][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:17,163][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:22,165][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:27,167][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:32,168][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:37,169][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:42,170][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:47,171][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:52,174][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:27:57,176][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:02,177][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:07,178][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:12,179][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:17,181][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:22,182][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:27,183][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:32,184][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:37,187][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:42,190][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:47,191][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:52,194][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:28:57,195][  INFO][PID:1636457] Checking for cancel request
[2025-07-15 21:29:00,362][  INFO][PID:1636457] Downloading results from builder
[2025-07-15 21:29:00,363][  INFO][PID:1636457] rsyncing of mockbuilder@44.202.70.43:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl started
[2025-07-15 21:29:00,364][  INFO][PID:1636457] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.202.70.43:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/ &> /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/build-09276940.rsync.log
[2025-07-15 21:29:00,947][  INFO][PID:1636457] rsyncing finished.
[2025-07-15 21:29:00,948][  INFO][PID:1636457] Releasing VM back to pool
[2025-07-15 21:29:00,960][  INFO][PID:1636457] Searching for 'success' file in resultdir
[2025-07-15 21:29:00,961][ ERROR][PID:1636457] Build failed: Backend process error: No success file => build failure
[2025-07-15 21:29:00,962][  INFO][PID:1636457] Finished build: id=9276940 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/lecris/cmake-ninja chroot=fedora-rawhide-x86_64 
[2025-07-15 21:29:00,964][ ERROR][PID:1636457] Backend process error: No success file => build failure
[2025-07-15 21:29:00,965][  INFO][PID:1636457] Worker failed build, took 814.376692533493
[2025-07-15 21:29:00,965][  INFO][PID:1636457] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "lecris",
            "project_name": "cmake-ninja",
            "project_dirname": "cmake-ninja",
            "submitter": "lecris",
            "ended_on": 1752614940.965272,
            "started_on": 1752614126.5885794,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9276940-fedora-rawhide-x86_64",
            "build_id": 9276940,
            "package_name": "mathgl",
            "package_version": "8.0.1-21",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lecris/cmake-ninja/mathgl",
            "git_hash": "1ebb96f32371fe11bd3c51e7b96ea67b1f1619c2",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "lecris/cmake-ninja--lecris",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-ninja/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/lecris/cmake-ninja",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lecris/cmake-ninja",
            "result_dir": "09276940-mathgl",
            "built_packages": "",
            "id": 9276940,
            "mockchain_macros": {
                "copr_username": "lecris",
                "copr_projectname": "cmake-ninja",
                "vendor": "Fedora Project COPR (lecris/cmake-ninja)"
            }
        }
    ]
}
[2025-07-15 21:29:00,992][  INFO][PID:1636457] Sending fedora-messaging bus message in build.end
[2025-07-15 21:29:01,019][  INFO][PID:1636457] Compressing /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/builder-live.log by gzip
[2025-07-15 21:29:01,020][  INFO][PID:1636457] Running command 'gzip /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/builder-live.log' as PID 1967940
[2025-07-15 21:29:01,064][  INFO][PID:1636457] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/builder-live.log)
[2025-07-15 21:29:01,065][  INFO][PID:1636457] Compressing /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/backend.log by gzip
[2025-07-15 21:29:01,066][  INFO][PID:1636457] Running command 'gzip /var/lib/copr/public_html/results/lecris/cmake-ninja/fedora-rawhide-x86_64/09276940-mathgl/backend.log' as PID 1967945