[2026-01-12 11:18:31,387][  INFO][PID:3182797] Marking build as starting
[2026-01-12 11:18:31,472][  INFO][PID:3182797] Checking that /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 11:18:31,563][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:18:31,564][  INFO][PID:3182797] VM allocation process starts
[2026-01-12 11:18:31,582][  INFO][PID:3182797] Trying to allocate VM: ResallocHost, ticket_id=775029, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:18:34,610][  INFO][PID:3182797] Allocated host ResallocHost, ticket_id=775029, hostname=44.210.150.29, name=aws_x86_64_normal_prod_01138061_20260112_110810, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:18:34,611][  INFO][PID:3182797] Allocating ssh connection to builder
[2026-01-12 11:18:34,612][  INFO][PID:3182797] Checking that builder machine is OK
[2026-01-12 11:18:34,974][  INFO][PID:3182797] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 11:18:35,220][  INFO][PID:3182797] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 11:18:35,224][  INFO][PID:3182797] Filling build.info file with builder info
[2026-01-12 11:18:35,228][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:18:35,232][  INFO][PID:3182797] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "ryanabx",
            "project_name": "cosmic-epoch",
            "project_dirname": "cosmic-epoch",
            "submitter": "ryanabx",
            "ended_on": null,
            "started_on": 1768216715.2319732,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10000584-fedora-rawhide-x86_64",
            "build_id": 10000584,
            "package_name": "pop-launcher",
            "package_version": "1:1.2.7^git20251001.eead361-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ryanabx/cosmic-epoch/pop-launcher",
            "git_hash": "fbfd9f6cdc00a4268cec1bbca750bd3e4c6b3514",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "ryanabx/cosmic-epoch--ryanabx",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": true,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/ryanabx/cosmic-epoch",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch",
            "result_dir": "10000584-pop-launcher",
            "built_packages": "",
            "id": 10000584,
            "mockchain_macros": {
                "copr_username": "ryanabx",
                "copr_projectname": "cosmic-epoch",
                "vendor": "Fedora Project COPR (ryanabx/cosmic-epoch)"
            }
        }
    ]
}
[2026-01-12 11:18:35,335][  INFO][PID:3182797] Sending fedora-messaging bus message in build.start
[2026-01-12 11:18:37,754][  INFO][PID:3182797] Sending fedora-messaging bus message in chroot.start
[2026-01-12 11:18:37,810][  INFO][PID:3182797] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000584-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 11:18:38,228][  INFO][PID:3182797] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.210.150.29' (ED25519) to the list of known hosts.


[2026-01-12 11:18:38,233][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 1
[2026-01-12 11:18:38,239][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.210.150.29 copr-rpmbuild-log
[2026-01-12 11:18:43,267][  INFO][PID:3182797] Periodic builder liveness probe: alive
[2026-01-12 11:18:43,268][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:18:48,269][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:18:53,270][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:18:58,271][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:03,272][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:08,274][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:13,275][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:18,279][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:23,281][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:28,282][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:33,283][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:38,295][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:43,299][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:48,300][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:53,310][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:19:58,314][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:03,326][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:08,328][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:13,330][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:18,333][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:23,335][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:28,339][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:33,340][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:38,341][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:43,342][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:48,344][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:53,347][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:20:58,352][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:03,355][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:08,365][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:13,371][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:18,378][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:23,379][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:28,382][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:33,384][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:38,400][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:43,404][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:48,406][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:53,411][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:21:58,414][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:03,416][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:08,419][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:13,420][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:18,422][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:23,424][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:28,425][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:33,427][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:38,429][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:43,431][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:48,434][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:53,435][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:22:58,437][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:03,439][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:08,441][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:13,443][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:18,444][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:23,445][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:28,448][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:33,449][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:38,450][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:43,452][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:48,454][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:53,459][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:23:58,460][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:03,465][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:08,467][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:13,469][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:18,470][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:23,471][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:28,473][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:33,474][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:38,475][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:43,476][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:48,477][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:53,479][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:24:58,480][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:03,481][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:08,482][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:13,483][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:18,484][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:23,485][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:28,486][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:33,487][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:38,488][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:43,489][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:48,490][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:53,491][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:25:58,492][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:03,493][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:08,494][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:13,495][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:18,496][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:23,497][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:28,498][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:33,499][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:38,500][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:43,501][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:48,503][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:53,504][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:26:58,505][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:03,506][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:08,507][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:13,508][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:18,509][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:23,510][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:28,518][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:33,519][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:38,520][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:43,521][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:48,522][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:53,523][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:27:58,525][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:03,526][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:08,529][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:13,530][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:18,532][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:23,533][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:28,534][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:33,535][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:38,536][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:43,545][  INFO][PID:3182797] Periodic builder liveness probe: alive
[2026-01-12 11:28:43,546][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:48,547][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:53,548][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:28:58,549][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:03,550][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:08,551][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:13,552][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:18,554][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:23,557][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:28,558][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:33,559][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:38,559][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:43,560][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:48,561][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:53,562][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:29:58,563][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:03,564][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:08,565][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:13,567][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:18,568][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:23,568][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:28,569][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:33,570][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:38,571][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:43,572][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:48,573][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:53,582][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:30:58,583][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:03,584][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:08,585][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:13,586][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:18,587][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:23,588][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:28,588][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:33,596][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:38,601][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:43,602][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:48,603][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:53,604][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:31:58,605][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:03,606][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:08,607][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:13,609][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:18,611][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:23,612][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:28,613][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:33,614][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:38,616][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:43,619][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:43,756][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:32:48,619][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:53,621][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:32:53,757][WARNING][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:32:58,622][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:03,623][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:03,758][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 2
[2026-01-12 11:33:03,760][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.210.150.29 copr-rpmbuild-log
[2026-01-12 11:33:08,624][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:13,626][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:18,627][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:23,628][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:28,631][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:33,632][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:38,633][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:43,634][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:48,635][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:53,636][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:33:58,638][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:03,639][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:03,829][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:34:08,640][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:13,641][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:13,834][WARNING][PID:3182797] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:34:18,643][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:23,649][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:23,835][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 3
[2026-01-12 11:34:23,836][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.210.150.29 copr-rpmbuild-log
[2026-01-12 11:34:28,650][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:33,651][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:38,652][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:43,653][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:48,655][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:53,656][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:34:58,657][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:03,659][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:08,661][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:13,662][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:18,663][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:23,664][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:23,903][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:35:28,665][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:33,666][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:33,910][WARNING][PID:3182797] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:35:38,667][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:43,668][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:43,911][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 4
[2026-01-12 11:35:43,912][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.210.150.29 copr-rpmbuild-log
[2026-01-12 11:35:48,669][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:53,670][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:35:58,671][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:03,672][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:08,673][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:13,674][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:18,675][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:23,676][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:28,677][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:33,678][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:38,679][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:43,680][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:43,981][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:36:48,681][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:53,682][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:53,998][ ERROR][PID:3182797] Giving up for unstable SSH, failures: 1768217573.7572024, 1768217653.8343253, 1768217733.9102595, 1768217813.9982147
[2026-01-12 11:36:53,999][ ERROR][PID:3182797] Re-try request for task on 'ResallocHost, ticket_id=775029, hostname=44.210.150.29, name=aws_x86_64_normal_prod_01138061_20260112_110810, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 11:36:53,999][  INFO][PID:3182797] Releasing VM back to pool
[2026-01-12 11:36:54,009][  INFO][PID:3182797] Retry #1 (on other host)
[2026-01-12 11:36:54,010][  INFO][PID:3182797] Checking that /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 11:36:54,011][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:36:54,011][  INFO][PID:3182797] VM allocation process starts
[2026-01-12 11:36:54,043][  INFO][PID:3182797] Trying to allocate VM: ResallocHost, ticket_id=775330, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:36:57,087][  INFO][PID:3182797] Allocated host ResallocHost, ticket_id=775330, hostname=98.92.10.165, name=aws_x86_64_spot_prod_01138612_20260112_113201, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:36:57,686][  INFO][PID:3182797] Allocating ssh connection to builder
[2026-01-12 11:36:57,687][  INFO][PID:3182797] Checking that builder machine is OK
[2026-01-12 11:37:57,755][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 98.92.10.165 port 22: Connection timed out

[2026-01-12 11:38:07,777][ ERROR][PID:3182797] Re-try request for task on 'ResallocHost, ticket_id=775330, hostname=98.92.10.165, name=aws_x86_64_spot_prod_01138612_20260112_113201, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2026-01-12 11:38:07,778][  INFO][PID:3182797] Releasing VM back to pool
[2026-01-12 11:38:07,787][  INFO][PID:3182797] Retry #2 (on other host)
[2026-01-12 11:38:07,788][  INFO][PID:3182797] Checking that /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 11:38:07,788][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:07,789][  INFO][PID:3182797] VM allocation process starts
[2026-01-12 11:38:07,838][  INFO][PID:3182797] Trying to allocate VM: ResallocHost, ticket_id=775358, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:38:10,897][  INFO][PID:3182797] Allocated host ResallocHost, ticket_id=775358, hostname=34.201.63.150, name=aws_x86_64_spot_prod_01138750_20260112_113517, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:38:10,897][  INFO][PID:3182797] Allocating ssh connection to builder
[2026-01-12 11:38:10,898][  INFO][PID:3182797] Checking that builder machine is OK
[2026-01-12 11:38:11,229][  INFO][PID:3182797] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 11:38:11,426][  INFO][PID:3182797] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 11:38:11,427][  INFO][PID:3182797] Filling build.info file with builder info
[2026-01-12 11:38:11,451][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:11,452][  INFO][PID:3182797] Not re-notifying FE and msg buses for the new host.
[2026-01-12 11:38:11,452][  INFO][PID:3182797] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000584-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 11:38:11,770][  INFO][PID:3182797] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '34.201.63.150' (ED25519) to the list of known hosts.


[2026-01-12 11:38:12,338][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 1
[2026-01-12 11:38:12,339][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.63.150 copr-rpmbuild-log
[2026-01-12 11:38:17,338][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:22,339][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:27,340][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:32,341][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:37,342][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:42,342][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:47,348][  INFO][PID:3182797] Periodic builder liveness probe: alive
[2026-01-12 11:38:47,348][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:52,349][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:38:57,350][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:02,351][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:07,352][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:12,353][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:17,354][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:22,355][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:27,356][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:32,358][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:37,359][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:42,360][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:47,363][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:52,364][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:39:57,366][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:02,367][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:07,368][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:12,369][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:17,370][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:22,371][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:27,371][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:32,372][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:37,373][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:42,374][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:47,375][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:52,376][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:40:57,377][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:02,378][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:07,379][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:12,380][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:17,381][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:22,450][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:27,454][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:32,461][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:37,463][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:42,463][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:47,464][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:52,465][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:41:57,466][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:02,467][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:07,468][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:12,469][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:17,470][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:22,470][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:27,471][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:32,472][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:37,473][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:42,474][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:47,475][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:52,476][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:42:57,477][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:02,478][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:07,479][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:12,481][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:17,482][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:22,483][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:27,485][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:32,486][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:37,487][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:42,488][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:47,489][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:52,490][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:43:57,490][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:02,491][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:07,492][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:12,493][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:17,494][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:22,495][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:27,496][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:32,497][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:37,498][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:42,499][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:47,500][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:52,501][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:44:57,502][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:02,512][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:07,513][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:12,515][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:17,516][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:22,517][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:27,632][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:32,634][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:37,642][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:42,644][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:47,645][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:52,657][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:45:57,658][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:02,659][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:07,661][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:12,662][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:17,663][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:22,748][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:27,752][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:32,754][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:37,802][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:42,820][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:47,824][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:52,829][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:46:57,833][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:02,836][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:07,868][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:12,869][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:18,013][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:23,027][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:28,028][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:33,029][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:38,030][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:43,034][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:48,037][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:53,038][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:47:58,039][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:03,040][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:08,041][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:13,074][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:18,084][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:23,090][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:28,093][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:33,095][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:38,099][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:43,105][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:48,118][  INFO][PID:3182797] Periodic builder liveness probe: alive
[2026-01-12 11:48:48,119][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:53,121][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:48:58,124][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:03,126][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:08,128][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:13,129][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:18,130][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:23,131][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:28,133][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:33,134][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:38,178][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:43,180][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:48,182][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:53,183][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:49:58,260][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:03,261][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:08,262][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:13,263][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:18,264][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:23,283][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:28,286][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:33,287][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:38,365][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:43,366][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:48,367][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:53,368][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:50:58,370][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:03,371][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:08,372][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:13,426][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:18,427][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:23,429][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:28,431][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:33,436][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:38,440][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:43,443][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:48,449][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:53,462][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:51:58,463][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:03,582][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:08,583][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:13,584][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:18,585][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:23,676][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:28,773][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:33,775][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:38,824][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:43,825][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:48,826][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:53,827][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:52:58,828][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:03,829][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:08,830][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:13,832][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:18,833][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:23,834][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:28,835][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:33,837][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:38,838][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:43,839][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:48,843][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:53,849][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:53:57,325][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:53:58,852][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:03,855][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:07,333][WARNING][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:54:08,857][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:13,858][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:17,333][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 2
[2026-01-12 11:54:17,335][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.63.150 copr-rpmbuild-log
[2026-01-12 11:54:18,859][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:23,860][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:28,890][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:33,893][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:38,897][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:43,898][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:48,901][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:53,912][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:54:58,913][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:03,914][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:08,916][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:13,917][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:17,399][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:55:18,917][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:23,919][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:27,401][WARNING][PID:3182797] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:55:28,920][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:33,921][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:37,402][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 3
[2026-01-12 11:55:37,404][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.63.150 copr-rpmbuild-log
[2026-01-12 11:55:38,922][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:43,923][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:48,948][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:54,108][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:55:59,254][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:04,266][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:09,483][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:14,487][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:19,606][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:24,608][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:29,609][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:34,614][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:37,472][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:56:39,627][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:44,645][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:47,501][WARNING][PID:3182797] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:56:49,647][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:54,680][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:56:57,502][  INFO][PID:3182797] Downloading the builder-live.log file, attempt 4
[2026-01-12 11:56:57,503][  INFO][PID:3182797] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.63.150 copr-rpmbuild-log
[2026-01-12 11:56:59,683][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:04,690][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:09,720][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:14,733][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:19,734][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:24,735][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:29,741][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:34,751][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:39,753][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:44,837][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:49,838][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:54,839][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:57:57,570][ ERROR][PID:3182797] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:57:59,840][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:58:04,842][  INFO][PID:3182797] Checking for cancel request
[2026-01-12 11:58:07,574][ ERROR][PID:3182797] Giving up for unstable SSH, failures: 1768218847.3331814, 1768218927.4017508, 1768219007.501281, 1768219087.5748155
[2026-01-12 11:58:07,575][ ERROR][PID:3182797] Re-try request for task on 'ResallocHost, ticket_id=775358, hostname=34.201.63.150, name=aws_x86_64_spot_prod_01138750_20260112_113517, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 11:58:07,576][  INFO][PID:3182797] Releasing VM back to pool
[2026-01-12 11:58:07,592][ ERROR][PID:3182797] Backend process error: Three host tried without success: {'44.210.150.29', '34.201.63.150', '98.92.10.165'}
[2026-01-12 11:58:07,592][WARNING][PID:3182797] Switching not-finished job state to 'failed'
[2026-01-12 11:58:07,593][  INFO][PID:3182797] Worker failed build, took 1196.140695810318
[2026-01-12 11:58:07,598][  INFO][PID:3182797] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "ryanabx",
            "project_name": "cosmic-epoch",
            "project_dirname": "cosmic-epoch",
            "submitter": "ryanabx",
            "ended_on": 1768219087.5927339,
            "started_on": 1768217891.452038,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10000584-fedora-rawhide-x86_64",
            "build_id": 10000584,
            "package_name": "pop-launcher",
            "package_version": "1:1.2.7^git20251001.eead361-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/ryanabx/cosmic-epoch/pop-launcher",
            "git_hash": "fbfd9f6cdc00a4268cec1bbca750bd3e4c6b3514",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "ryanabx/cosmic-epoch--ryanabx",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": true,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/ryanabx/cosmic-epoch",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/ryanabx/cosmic-epoch",
            "result_dir": "10000584-pop-launcher",
            "built_packages": "",
            "id": 10000584,
            "mockchain_macros": {
                "copr_username": "ryanabx",
                "copr_projectname": "cosmic-epoch",
                "vendor": "Fedora Project COPR (ryanabx/cosmic-epoch)"
            }
        }
    ]
}
[2026-01-12 11:58:07,656][  INFO][PID:3182797] Sending fedora-messaging bus message in build.end
[2026-01-12 11:58:07,712][  INFO][PID:3182797] Compressing /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/10000584-pop-launcher/builder-live.log by gzip
[2026-01-12 11:58:07,714][  INFO][PID:3182797] Running command 'gzip /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/10000584-pop-launcher/builder-live.log' as PID 3448126
[2026-01-12 11:58:07,746][  INFO][PID:3182797] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/10000584-pop-launcher/builder-live.log)
[2026-01-12 11:58:07,747][  INFO][PID:3182797] Compressing /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/10000584-pop-launcher/backend.log by gzip
[2026-01-12 11:58:07,748][  INFO][PID:3182797] Running command 'gzip /var/lib/copr/public_html/results/ryanabx/cosmic-epoch/fedora-rawhide-x86_64/10000584-pop-launcher/backend.log' as PID 3448134