[2025-07-12 02:37:52,429][  INFO][PID:1879664] Marking build as starting
[2025-07-12 02:37:52,451][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:37:52,452][  INFO][PID:1879664] VM allocation process starts
[2025-07-12 02:37:52,484][  INFO][PID:1879664] Trying to allocate VM: ResallocHost, ticket_id=3902298, requested_tags=['copr_builder', 'arch_noarch']
[2025-07-12 02:37:55,550][  INFO][PID:1879664] Allocated host ResallocHost, ticket_id=3902298, hostname=2620:52:3:1:dead:beef:cafe:c29a, name=copr_p09_01_prod_05307823_20250712_021609, requested_tags=['copr_builder', 'arch_noarch']
[2025-07-12 02:37:55,550][  INFO][PID:1879664] Allocating ssh connection to builder
[2025-07-12 02:37:55,552][  INFO][PID:1879664] Checking that builder machine is OK
[2025-07-12 02:37:56,286][  INFO][PID:1879664] Running remote command: copr-builder-ready srpm-builds
[2025-07-12 02:37:56,709][  INFO][PID:1879664] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-07-12 02:37:56,710][  INFO][PID:1879664] Filling build.info file with builder info
[2025-07-12 02:37:56,718][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:37:56,724][  INFO][PID:1879664] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20250712",
            "project_dirname": "llvm-snapshots-big-merge-20250712",
            "submitter": "kkleine",
            "ended_on": null,
            "started_on": 1752287876.7238846,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9268315",
            "build_id": 9268315,
            "package_name": "llvm",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 8,
            "source_json": "{\"type\": \"git\", \"clone_url\": \"https://src.fedoraproject.org/rpms/llvm.git\", \"subdirectory\": \"\", \"committish\": \"rawhide\", \"spec\": \"llvm.spec\", \"srpm_build_method\": \"make_srpm\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20250712--kkleine",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712",
            "result_dir": "09268315",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 9268315,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20250712",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20250712)"
            }
        }
    ]
}
[2025-07-12 02:37:56,773][  INFO][PID:1879664] Sending fedora-messaging bus message in build.start
[2025-07-12 02:37:58,860][  INFO][PID:1879664] Sending fedora-messaging bus message in chroot.start
[2025-07-12 02:37:58,891][  INFO][PID:1879664] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9268315 --detached
[2025-07-12 02:37:59,562][  INFO][PID:1879664] Downloading the builder-live.log file, attempt 1
[2025-07-12 02:37:59,570][  INFO][PID:1879664] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-12 02:38:04,569][  INFO][PID:1879664] Periodic builder liveness probe: alive
[2025-07-12 02:38:04,569][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:09,570][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:14,571][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:19,572][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:24,573][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:29,574][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:34,575][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:39,577][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:44,578][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:49,579][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:54,581][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:38:59,583][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:04,584][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:09,586][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:14,592][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:19,605][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:24,608][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:29,611][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:34,612][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:39,613][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:44,615][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:49,616][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:54,617][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:39:59,619][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:00,139][ ERROR][PID:1879664] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-12 02:40:04,620][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:09,622][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:10,139][WARNING][PID:1879664] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-12 02:40:14,623][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:19,625][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:20,140][  INFO][PID:1879664] Downloading the builder-live.log file, attempt 2
[2025-07-12 02:40:20,142][  INFO][PID:1879664] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-12 02:40:24,626][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:29,628][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:34,631][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:39,633][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:44,636][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:49,637][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:54,638][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:40:59,639][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:04,641][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:09,642][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:14,644][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:19,645][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:20,211][ ERROR][PID:1879664] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-12 02:41:24,647][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:29,648][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:30,219][WARNING][PID:1879664] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-12 02:41:34,650][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:39,652][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:40,220][  INFO][PID:1879664] Downloading the builder-live.log file, attempt 3
[2025-07-12 02:41:40,222][  INFO][PID:1879664] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-12 02:41:44,654][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:49,655][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:54,657][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:41:59,658][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:04,660][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:09,660][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:14,662][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:19,663][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:24,664][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:29,665][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:34,666][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:39,668][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:40,290][ ERROR][PID:1879664] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-12 02:42:44,669][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:49,671][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:50,302][WARNING][PID:1879664] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-12 02:42:54,672][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:42:59,673][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:00,303][  INFO][PID:1879664] Downloading the builder-live.log file, attempt 4
[2025-07-12 02:43:00,305][  INFO][PID:1879664] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29a copr-rpmbuild-log
[2025-07-12 02:43:04,674][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:09,676][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:14,677][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:19,678][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:24,679][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:29,680][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:34,681][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:39,682][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:44,683][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:49,684][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:54,685][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:43:59,687][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:00,373][ ERROR][PID:1879664] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-12 02:44:04,688][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:09,689][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:10,389][ ERROR][PID:1879664] Giving up for unstable SSH, failures: 1752288010.1399033, 1752288090.2197828, 1752288170.3026912, 1752288250.3891973
[2025-07-12 02:44:10,390][ ERROR][PID:1879664] Re-try request for task on 'ResallocHost, ticket_id=3902298, hostname=2620:52:3:1:dead:beef:cafe:c29a, name=copr_p09_01_prod_05307823_20250712_021609, requested_tags=['copr_builder', 'arch_noarch']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-12 02:44:10,390][  INFO][PID:1879664] Releasing VM back to pool
[2025-07-12 02:44:10,399][  INFO][PID:1879664] Retry #1 (on other host)
[2025-07-12 02:44:10,399][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:10,400][  INFO][PID:1879664] VM allocation process starts
[2025-07-12 02:44:10,449][  INFO][PID:1879664] Trying to allocate VM: ResallocHost, ticket_id=3902352, requested_tags=['copr_builder', 'arch_noarch']
[2025-07-12 02:44:13,499][  INFO][PID:1879664] Allocated host ResallocHost, ticket_id=3902352, hostname=3.91.190.180, name=aws_aarch64_normalreserved_prod_05308160_20250712_023819, requested_tags=['copr_builder', 'arch_noarch']
[2025-07-12 02:44:13,499][  INFO][PID:1879664] Allocating ssh connection to builder
[2025-07-12 02:44:13,499][  INFO][PID:1879664] Checking that builder machine is OK
[2025-07-12 02:44:13,854][  INFO][PID:1879664] Running remote command: copr-builder-ready srpm-builds
[2025-07-12 02:44:14,073][  INFO][PID:1879664] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-07-12 02:44:14,073][  INFO][PID:1879664] Filling build.info file with builder info
[2025-07-12 02:44:14,089][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:14,090][  INFO][PID:1879664] Not re-notifying FE and msg buses for the new host.
[2025-07-12 02:44:14,090][  INFO][PID:1879664] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9268315 --detached
[2025-07-12 02:44:14,491][  INFO][PID:1879664] Downloading the builder-live.log file, attempt 1
[2025-07-12 02:44:14,492][  INFO][PID:1879664] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.190.180 copr-rpmbuild-log
[2025-07-12 02:44:19,491][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:24,493][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:29,494][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:34,495][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:39,496][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:44,497][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:49,498][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:54,499][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:44:59,500][  INFO][PID:1879664] Checking for cancel request
[2025-07-12 02:45:03,914][  INFO][PID:1879664] Downloading results from builder
[2025-07-12 02:45:03,914][  INFO][PID:1879664] rsyncing of mockbuilder@3.91.190.180:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315 started
[2025-07-12 02:45:03,915][  INFO][PID:1879664] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.91.190.180:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/build-09268315.rsync.log
[2025-07-12 02:45:05,312][  INFO][PID:1879664] rsyncing finished.
[2025-07-12 02:45:05,312][  INFO][PID:1879664] Releasing VM back to pool
[2025-07-12 02:45:05,322][  INFO][PID:1879664] Searching for 'success' file in resultdir
[2025-07-12 02:45:05,323][  INFO][PID:1879664] Getting build details
[2025-07-12 02:45:05,323][  INFO][PID:1879664] Retrieving SRPM info from /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315
[2025-07-12 02:45:05,324][  INFO][PID:1879664] SRPM URL: https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/llvm-21.0.0~pre20250712.g102c15ad28d3c3-1.src.rpm
[2025-07-12 02:45:05,324][  INFO][PID:1879664] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/llvm-21.0.0~pre20250712.g102c15ad28d3c3-1.src.rpm', 'pkg_name': 'llvm', 'pkg_version': '20.1.8-1'}
[2025-07-12 02:45:05,325][  INFO][PID:1879664] Finished build: id=9268315 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712 chroot=srpm-builds 
[2025-07-12 02:45:05,326][  INFO][PID:1879664] Worker succeeded build, took 51.236467599868774
[2025-07-12 02:45:05,327][  INFO][PID:1879664] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20250712",
            "project_dirname": "llvm-snapshots-big-merge-20250712",
            "submitter": "kkleine",
            "ended_on": 1752288305.3265514,
            "started_on": 1752288254.0900838,
            "submitted_on": null,
            "status": 1,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9268315",
            "build_id": 9268315,
            "package_name": "llvm",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 8,
            "source_json": "{\"type\": \"git\", \"clone_url\": \"https://src.fedoraproject.org/rpms/llvm.git\", \"subdirectory\": \"\", \"committish\": \"rawhide\", \"spec\": \"llvm.spec\", \"srpm_build_method\": \"make_srpm\"}",
            "pkg_name": "llvm",
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/llvm-21.0.0~pre20250712.g102c15ad28d3c3-1.src.rpm",
            "uses_devel_repo": null,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20250712--kkleine",
            "results": {
                "name": "llvm",
                "epoch": null,
                "version": "20.1.8",
                "release": "1",
                "exclusivearch": [],
                "excludearch": []
            },
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712",
            "result_dir": "09268315",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "pkg_version": "20.1.8-1",
            "id": 9268315,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20250712",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20250712)"
            }
        }
    ]
}
[2025-07-12 02:45:05,354][  INFO][PID:1879664] Sending fedora-messaging bus message in build.end
[2025-07-12 02:45:05,379][  INFO][PID:1879664] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/builder-live.log by gzip
[2025-07-12 02:45:05,380][  INFO][PID:1879664] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/builder-live.log' as PID 1924731
[2025-07-12 02:45:05,384][  INFO][PID:1879664] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/builder-live.log)
[2025-07-12 02:45:05,385][  INFO][PID:1879664] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/backend.log by gzip
[2025-07-12 02:45:05,386][  INFO][PID:1879664] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250712/srpm-builds/09268315/backend.log' as PID 1924732