[2026-01-12 12:13:03,205][  INFO][PID:3586211] Marking build as starting
[2026-01-12 12:13:03,255][  INFO][PID:3586211] Checking that /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 12:13:03,256][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:03,258][  INFO][PID:3586211] VM allocation process starts
[2026-01-12 12:13:03,277][  INFO][PID:3586211] Trying to allocate VM: ResallocHost, ticket_id=775948, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 12:13:06,293][  INFO][PID:3586211] Allocated host ResallocHost, ticket_id=775948, hostname=13.221.49.7, name=aws_x86_64_spot_prod_01141022_20260112_120926, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 12:13:06,293][  INFO][PID:3586211] Allocating ssh connection to builder
[2026-01-12 12:13:06,294][  INFO][PID:3586211] Checking that builder machine is OK
[2026-01-12 12:13:06,625][  INFO][PID:3586211] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 12:13:06,839][  INFO][PID:3586211] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 12:13:06,839][  INFO][PID:3586211] Filling build.info file with builder info
[2026-01-12 12:13:06,971][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:06,972][  INFO][PID:3586211] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "pbrobinson",
            "project_name": "test-aiwg-packages",
            "project_dirname": "test-aiwg-packages",
            "submitter": "pbrobinson",
            "ended_on": null,
            "started_on": 1768219986.9727504,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10000757-fedora-rawhide-x86_64",
            "build_id": 10000757,
            "package_name": "onnx",
            "package_version": "1.20.1-0",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pbrobinson/test-aiwg-packages/onnx",
            "git_hash": "cdaf057a01b608f185d5f8479ffa215460b3d7d2",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "pbrobinson/test-aiwg-packages--pbrobinson",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pbrobinson/test-aiwg-packages",
            "result_dir": "10000757-onnx",
            "built_packages": "",
            "id": 10000757,
            "mockchain_macros": {
                "copr_username": "pbrobinson",
                "copr_projectname": "test-aiwg-packages",
                "vendor": "Fedora Project COPR (pbrobinson/test-aiwg-packages)"
            }
        }
    ]
}
[2026-01-12 12:13:07,020][  INFO][PID:3586211] Sending fedora-messaging bus message in build.start
[2026-01-12 12:13:07,535][  INFO][PID:3586211] Sending fedora-messaging bus message in chroot.start
[2026-01-12 12:13:07,562][  INFO][PID:3586211] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000757-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 12:13:07,876][  INFO][PID:3586211] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '13.221.49.7' (ED25519) to the list of known hosts.


[2026-01-12 12:13:07,876][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 1
[2026-01-12 12:13:07,878][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.49.7 copr-rpmbuild-log
[2026-01-12 12:13:12,917][  INFO][PID:3586211] Periodic builder liveness probe: alive
[2026-01-12 12:13:12,917][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:17,918][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:22,919][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:27,920][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:32,921][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:34,614][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:13:37,922][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:42,923][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:44,615][WARNING][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 12:13:47,925][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:52,927][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:13:54,616][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 2
[2026-01-12 12:13:54,618][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.49.7 copr-rpmbuild-log
[2026-01-12 12:13:57,928][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:02,929][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:07,931][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:12,932][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:17,933][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:22,934][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:27,935][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:32,936][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:37,937][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:42,938][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:47,962][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:52,964][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:14:54,628][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:14:57,965][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:02,966][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:04,640][WARNING][PID:3586211] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 12:15:07,967][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:12,968][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:14,641][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 3
[2026-01-12 12:15:14,643][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.49.7 copr-rpmbuild-log
[2026-01-12 12:15:17,969][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:22,970][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:27,971][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:32,973][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:37,974][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:42,974][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:47,975][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:52,976][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:15:57,977][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:02,978][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:07,980][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:12,982][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:14,711][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:16:17,983][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:22,984][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:24,734][WARNING][PID:3586211] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 12:16:27,985][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:32,988][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:34,735][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 4
[2026-01-12 12:16:34,736][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.49.7 copr-rpmbuild-log
[2026-01-12 12:16:37,990][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:42,991][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:47,992][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:52,994][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:16:57,995][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:02,996][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:07,997][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:12,998][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:17,999][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:23,000][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:28,002][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:33,003][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:34,806][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:17:38,004][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:43,006][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:44,813][ ERROR][PID:3586211] Giving up for unstable SSH, failures: 1768220024.6153884, 1768220104.6407564, 1768220184.7345428, 1768220264.8137324
[2026-01-12 12:17:44,814][ ERROR][PID:3586211] Re-try request for task on 'ResallocHost, ticket_id=775948, hostname=13.221.49.7, name=aws_x86_64_spot_prod_01141022_20260112_120926, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 12:17:44,815][  INFO][PID:3586211] Releasing VM back to pool
[2026-01-12 12:17:44,830][  INFO][PID:3586211] Retry #1 (on other host)
[2026-01-12 12:17:44,831][  INFO][PID:3586211] Checking that /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 12:17:44,832][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:44,833][  INFO][PID:3586211] VM allocation process starts
[2026-01-12 12:17:44,846][  INFO][PID:3586211] Trying to allocate VM: ResallocHost, ticket_id=775994, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 12:17:47,866][  INFO][PID:3586211] Allocated host ResallocHost, ticket_id=775994, hostname=3.236.239.192, name=aws_x86_64_spot_prod_01141323_20260112_121444, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 12:17:47,867][  INFO][PID:3586211] Allocating ssh connection to builder
[2026-01-12 12:17:47,867][  INFO][PID:3586211] Checking that builder machine is OK
[2026-01-12 12:17:48,193][  INFO][PID:3586211] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 12:17:48,398][  INFO][PID:3586211] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 12:17:48,399][  INFO][PID:3586211] Filling build.info file with builder info
[2026-01-12 12:17:48,406][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:48,408][  INFO][PID:3586211] Not re-notifying FE and msg buses for the new host.
[2026-01-12 12:17:48,409][  INFO][PID:3586211] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000757-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 12:17:48,731][  INFO][PID:3586211] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.236.239.192' (ED25519) to the list of known hosts.


[2026-01-12 12:17:48,733][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 1
[2026-01-12 12:17:48,738][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.192 copr-rpmbuild-log
[2026-01-12 12:17:53,733][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:17:58,734][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:03,736][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:08,737][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:13,738][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:18,739][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:23,743][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:28,744][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:33,747][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:38,749][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:43,750][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:48,751][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:53,752][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:18:55,658][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:18:58,756][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:03,758][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:05,660][WARNING][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 12:19:08,762][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:13,775][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:15,661][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 2
[2026-01-12 12:19:15,663][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.192 copr-rpmbuild-log
[2026-01-12 12:19:18,781][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:23,783][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:28,784][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:33,785][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:38,786][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:43,787][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:48,788][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:53,794][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:19:58,796][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:03,797][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:08,798][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:13,802][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:15,730][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:20:18,809][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:23,811][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:25,749][WARNING][PID:3586211] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 12:20:28,812][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:33,813][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:35,750][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 3
[2026-01-12 12:20:35,752][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.192 copr-rpmbuild-log
[2026-01-12 12:20:38,819][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:43,820][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:48,821][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:53,823][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:20:58,824][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:03,825][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:08,826][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:13,827][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:18,829][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:23,829][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:28,831][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:33,832][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:35,820][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:21:38,833][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:43,834][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:45,826][WARNING][PID:3586211] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 12:21:48,835][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:53,836][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:21:55,826][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 4
[2026-01-12 12:21:55,828][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.239.192 copr-rpmbuild-log
[2026-01-12 12:21:58,837][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:03,838][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:08,840][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:13,841][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:18,842][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:23,844][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:28,851][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:33,852][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:38,853][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:43,854][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:48,855][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:53,856][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:22:55,889][ ERROR][PID:3586211] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 12:22:58,857][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:03,858][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:05,898][ ERROR][PID:3586211] Giving up for unstable SSH, failures: 1768220345.6608207, 1768220425.7498748, 1768220505.8261292, 1768220585.8987033
[2026-01-12 12:23:05,902][ ERROR][PID:3586211] Re-try request for task on 'ResallocHost, ticket_id=775994, hostname=3.236.239.192, name=aws_x86_64_spot_prod_01141323_20260112_121444, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 12:23:05,903][  INFO][PID:3586211] Releasing VM back to pool
[2026-01-12 12:23:05,918][  INFO][PID:3586211] Retry #2 (on other host)
[2026-01-12 12:23:05,919][  INFO][PID:3586211] Checking that /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 12:23:05,919][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:05,920][  INFO][PID:3586211] VM allocation process starts
[2026-01-12 12:23:05,934][  INFO][PID:3586211] Trying to allocate VM: ResallocHost, ticket_id=776074, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 12:23:08,960][  INFO][PID:3586211] Allocated host ResallocHost, ticket_id=776074, hostname=98.84.159.96, name=aws_x86_64_spot_prod_01141765_20260112_122121, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 12:23:08,962][  INFO][PID:3586211] Allocating ssh connection to builder
[2026-01-12 12:23:08,963][  INFO][PID:3586211] Checking that builder machine is OK
[2026-01-12 12:23:09,341][  INFO][PID:3586211] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 12:23:09,706][  INFO][PID:3586211] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 12:23:09,707][  INFO][PID:3586211] Filling build.info file with builder info
[2026-01-12 12:23:09,724][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:09,729][  INFO][PID:3586211] Not re-notifying FE and msg buses for the new host.
[2026-01-12 12:23:09,730][  INFO][PID:3586211] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000757-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 12:23:10,074][  INFO][PID:3586211] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '98.84.159.96' (ED25519) to the list of known hosts.


[2026-01-12 12:23:10,076][  INFO][PID:3586211] Downloading the builder-live.log file, attempt 1
[2026-01-12 12:23:10,079][  INFO][PID:3586211] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.84.159.96 copr-rpmbuild-log
[2026-01-12 12:23:15,091][  INFO][PID:3586211] Periodic builder liveness probe: alive
[2026-01-12 12:23:15,092][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:20,097][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:25,098][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:30,099][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:35,100][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:40,108][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:45,113][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:50,115][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:23:55,117][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:00,122][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:05,123][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:10,124][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:15,125][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:20,127][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:25,131][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:30,133][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:35,137][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:40,139][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:45,143][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:50,149][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:24:55,151][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:00,156][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:05,160][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:10,162][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:15,166][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:20,173][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:25,175][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:30,177][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:35,185][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:40,187][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:45,188][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:50,189][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:25:55,190][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:26:00,191][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:26:05,192][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:26:10,194][  INFO][PID:3586211] Checking for cancel request
[2026-01-12 12:26:10,971][  INFO][PID:3586211] Downloading results from builder
[2026-01-12 12:26:10,971][  INFO][PID:3586211] rsyncing of mockbuilder@98.84.159.96:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx started
[2026-01-12 12:26:10,972][  INFO][PID:3586211] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@98.84.159.96:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/ &> /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/build-10000757.rsync.log
[2026-01-12 12:26:11,400][  INFO][PID:3586211] rsyncing finished.
[2026-01-12 12:26:11,402][  INFO][PID:3586211] Releasing VM back to pool
[2026-01-12 12:26:11,443][  INFO][PID:3586211] Searching for 'success' file in resultdir
[2026-01-12 12:26:11,445][ ERROR][PID:3586211] Build failed: Backend process error: No success file => build failure
[2026-01-12 12:26:11,447][  INFO][PID:3586211] Finished build: id=10000757 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages chroot=fedora-rawhide-x86_64 
[2026-01-12 12:26:11,455][ ERROR][PID:3586211] Backend process error: No success file => build failure
[2026-01-12 12:26:11,456][  INFO][PID:3586211] Worker failed build, took 181.7271752357483
[2026-01-12 12:26:11,457][  INFO][PID:3586211] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "pbrobinson",
            "project_name": "test-aiwg-packages",
            "project_dirname": "test-aiwg-packages",
            "submitter": "pbrobinson",
            "ended_on": 1768220771.4563346,
            "started_on": 1768220589.7291594,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10000757-fedora-rawhide-x86_64",
            "build_id": 10000757,
            "package_name": "onnx",
            "package_version": "1.20.1-0",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pbrobinson/test-aiwg-packages/onnx",
            "git_hash": "cdaf057a01b608f185d5f8479ffa215460b3d7d2",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "pbrobinson/test-aiwg-packages--pbrobinson",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pbrobinson/test-aiwg-packages",
            "result_dir": "10000757-onnx",
            "built_packages": "",
            "id": 10000757,
            "mockchain_macros": {
                "copr_username": "pbrobinson",
                "copr_projectname": "test-aiwg-packages",
                "vendor": "Fedora Project COPR (pbrobinson/test-aiwg-packages)"
            }
        }
    ]
}
[2026-01-12 12:26:11,507][  INFO][PID:3586211] Sending fedora-messaging bus message in build.end
[2026-01-12 12:26:11,549][  INFO][PID:3586211] Compressing /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/builder-live.log by gzip
[2026-01-12 12:26:11,550][  INFO][PID:3586211] Running command 'gzip /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/builder-live.log' as PID 3684730
[2026-01-12 12:26:11,594][  INFO][PID:3586211] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/builder-live.log)
[2026-01-12 12:26:11,595][  INFO][PID:3586211] Compressing /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/backend.log by gzip
[2026-01-12 12:26:11,598][  INFO][PID:3586211] Running command 'gzip /var/lib/copr/public_html/results/pbrobinson/test-aiwg-packages/fedora-rawhide-x86_64/10000757-onnx/backend.log' as PID 3684737