[2025-07-10 14:58:37,273][  INFO][PID:3191111] Marking build as starting
[2025-07-10 14:58:37,310][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:58:37,312][  INFO][PID:3191111] VM allocation process starts
[2025-07-10 14:58:37,344][  INFO][PID:3191111] Trying to allocate VM: ResallocHost, ticket_id=3880509, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-10 14:58:42,350][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:58:47,354][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:58:49,405][  INFO][PID:3191111] Allocated host ResallocHost, ticket_id=3880509, hostname=3.239.231.159, name=aws_x86_64_spot_prod_05257470_20250710_145541, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-10 14:58:49,406][  INFO][PID:3191111] Allocating ssh connection to builder
[2025-07-10 14:58:49,406][  INFO][PID:3191111] Checking that builder machine is OK
[2025-07-10 14:58:49,715][  INFO][PID:3191111] Running remote command: copr-builder-ready srpm-builds
[2025-07-10 14:58:49,973][  INFO][PID:3191111] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-07-10 14:58:49,975][  INFO][PID:3191111] Filling build.info file with builder info
[2025-07-10 14:58:49,977][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:58:49,985][  INFO][PID:3191111] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "packit",
            "project_name": "packit-packit-2632",
            "project_dirname": "packit-packit-2632",
            "submitter": "packit",
            "ended_on": null,
            "started_on": 1752159529.9856079,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": "x86_64",
            "buildroot_pkgs": null,
            "task_id": "9263162",
            "build_id": 9263162,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 9,
            "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --pr-id 2632 --merge-pr --target-branch main --job-config-index 1 --merged-ref 'main' --no-create-symlinks https://github.com/packit/packit\\n\\n\", \"chroot\": \"fedora-42-x86_64\", \"builddeps\": \"packit python3-build hatch python3-hatch-vcs\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "packit/packit-packit-2632--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-42-x86_64/",
                    "id": "copr_packit_packit_stable",
                    "name": "Additional repo copr_packit_packit_stable"
                }
            ],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/packit/packit-packit-2632",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/packit-packit-2632",
            "result_dir": "09263162",
            "built_packages": "",
            "tags": [
                "arch_x86_64"
            ],
            "id": 9263162,
            "mockchain_macros": {
                "copr_username": "packit",
                "copr_projectname": "packit-packit-2632",
                "vendor": "Fedora Project COPR (packit/packit-packit-2632)"
            }
        }
    ]
}
[2025-07-10 14:58:50,021][  INFO][PID:3191111] Sending fedora-messaging bus message in build.start
[2025-07-10 14:58:51,544][  INFO][PID:3191111] Sending fedora-messaging bus message in chroot.start
[2025-07-10 14:58:51,606][  INFO][PID:3191111] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9263162 --detached
[2025-07-10 14:58:51,966][  INFO][PID:3191111] Downloading the builder-live.log file, attempt 1
[2025-07-10 14:58:51,983][  INFO][PID:3191111] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.231.159 copr-rpmbuild-log
[2025-07-10 14:58:56,997][  INFO][PID:3191111] Periodic builder liveness probe: alive
[2025-07-10 14:58:56,997][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:02,017][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:07,023][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:12,024][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:17,026][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:22,030][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:27,032][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:30,380][ ERROR][PID:3191111] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-10 14:59:32,036][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:37,042][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:40,390][WARNING][PID:3191111] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-10 14:59:42,047][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:47,060][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:50,391][  INFO][PID:3191111] Downloading the builder-live.log file, attempt 2
[2025-07-10 14:59:50,393][  INFO][PID:3191111] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.231.159 copr-rpmbuild-log
[2025-07-10 14:59:52,065][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 14:59:57,071][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:02,074][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:07,077][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:12,079][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:17,083][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:22,087][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:27,092][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:32,097][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:37,099][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:42,100][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:47,104][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:50,483][ ERROR][PID:3191111] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-10 15:00:52,115][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:00:57,117][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:00,487][WARNING][PID:3191111] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-10 15:01:02,120][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:07,125][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:10,492][  INFO][PID:3191111] Downloading the builder-live.log file, attempt 3
[2025-07-10 15:01:10,497][  INFO][PID:3191111] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.231.159 copr-rpmbuild-log
[2025-07-10 15:01:12,131][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:17,143][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:22,152][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:27,157][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:32,161][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:37,165][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:42,170][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:47,174][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:52,175][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:01:57,178][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:02,188][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:07,193][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:10,587][ ERROR][PID:3191111] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-10 15:02:12,196][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:17,198][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:20,603][WARNING][PID:3191111] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-10 15:02:22,203][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:27,206][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:30,610][  INFO][PID:3191111] Downloading the builder-live.log file, attempt 4
[2025-07-10 15:02:30,613][  INFO][PID:3191111] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.231.159 copr-rpmbuild-log
[2025-07-10 15:02:32,213][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:37,220][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:42,222][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:47,230][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:52,232][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:02:57,233][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:02,235][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:07,243][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:12,247][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:17,250][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:22,252][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:27,254][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:30,698][ ERROR][PID:3191111] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-10 15:03:32,256][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:37,257][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:40,700][ ERROR][PID:3191111] Giving up for unstable SSH, failures: 1752159580.3904924, 1752159660.4877691, 1752159740.6037447, 1752159820.7002044
[2025-07-10 15:03:40,702][ ERROR][PID:3191111] Re-try request for task on 'ResallocHost, ticket_id=3880509, hostname=3.239.231.159, name=aws_x86_64_spot_prod_05257470_20250710_145541, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-10 15:03:40,704][  INFO][PID:3191111] Releasing VM back to pool
[2025-07-10 15:03:40,729][  INFO][PID:3191111] Retry #1 (on other host)
[2025-07-10 15:03:40,733][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:40,734][  INFO][PID:3191111] VM allocation process starts
[2025-07-10 15:03:40,770][  INFO][PID:3191111] Trying to allocate VM: ResallocHost, ticket_id=3880573, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-10 15:03:45,777][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:50,778][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:03:55,785][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:00,793][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:05,794][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:10,796][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:15,800][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:20,804][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:25,813][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:30,815][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:35,817][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:40,820][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:45,828][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:48,949][  INFO][PID:3191111] Allocated host ResallocHost, ticket_id=3880573, hostname=54.162.232.181, name=aws_x86_64_normalreserved_prod_05257731_20250710_150204, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-10 15:04:48,950][  INFO][PID:3191111] Allocating ssh connection to builder
[2025-07-10 15:04:48,951][  INFO][PID:3191111] Checking that builder machine is OK
[2025-07-10 15:04:49,383][  INFO][PID:3191111] Running remote command: copr-builder-ready srpm-builds
[2025-07-10 15:04:49,609][  INFO][PID:3191111] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-07-10 15:04:49,611][  INFO][PID:3191111] Filling build.info file with builder info
[2025-07-10 15:04:49,613][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:49,614][  INFO][PID:3191111] Not re-notifying FE and msg buses for the new host.
[2025-07-10 15:04:49,615][  INFO][PID:3191111] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9263162 --detached
[2025-07-10 15:04:49,978][  INFO][PID:3191111] Downloading the builder-live.log file, attempt 1
[2025-07-10 15:04:49,983][  INFO][PID:3191111] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.162.232.181 copr-rpmbuild-log
[2025-07-10 15:04:54,982][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:04:59,991][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:04,997][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:10,006][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:15,008][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:20,013][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:25,015][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:30,019][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:35,020][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:40,024][  INFO][PID:3191111] Checking for cancel request
[2025-07-10 15:05:40,260][  INFO][PID:3191111] Downloading results from builder
[2025-07-10 15:05:40,261][  INFO][PID:3191111] rsyncing of mockbuilder@54.162.232.181:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162 started
[2025-07-10 15:05:40,263][  INFO][PID:3191111] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.162.232.181:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/ &> /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/build-09263162.rsync.log
[2025-07-10 15:05:40,733][  INFO][PID:3191111] rsyncing finished.
[2025-07-10 15:05:40,734][  INFO][PID:3191111] Releasing VM back to pool
[2025-07-10 15:05:40,755][  INFO][PID:3191111] Searching for 'success' file in resultdir
[2025-07-10 15:05:40,758][  INFO][PID:3191111] Getting build details
[2025-07-10 15:05:40,759][  INFO][PID:3191111] Retrieving SRPM info from /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162
[2025-07-10 15:05:40,763][  INFO][PID:3191111] SRPM URL: https://download.copr.fedorainfracloud.org/results/packit/packit-packit-2632/srpm-builds/09263162/packit-1.9.0.post1.dev7+g7ce12b95-1.20250710150535797465.pr2632.7.g7ce12b95.src.rpm
[2025-07-10 15:05:40,764][  INFO][PID:3191111] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/packit/packit-packit-2632/srpm-builds/09263162/packit-1.9.0.post1.dev7+g7ce12b95-1.20250710150535797465.pr2632.7.g7ce12b95.src.rpm', 'pkg_name': 'packit', 'pkg_version': '1.9.0.post1.dev7+g7ce12b95-1.20250710150535797465.pr2632.7.g7ce12b95'}
[2025-07-10 15:05:40,773][  INFO][PID:3191111] Finished build: id=9263162 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/packit/packit-packit-2632 chroot=srpm-builds 
[2025-07-10 15:05:40,781][  INFO][PID:3191111] Worker succeeded build, took 51.16723585128784
[2025-07-10 15:05:40,782][  INFO][PID:3191111] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "packit",
            "project_name": "packit-packit-2632",
            "project_dirname": "packit-packit-2632",
            "submitter": "packit",
            "ended_on": 1752159940.781872,
            "started_on": 1752159889.6146362,
            "submitted_on": null,
            "status": 1,
            "chroot": "srpm-builds",
            "arch": "x86_64",
            "buildroot_pkgs": null,
            "task_id": "9263162",
            "build_id": 9263162,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 9,
            "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --pr-id 2632 --merge-pr --target-branch main --job-config-index 1 --merged-ref 'main' --no-create-symlinks https://github.com/packit/packit\\n\\n\", \"chroot\": \"fedora-42-x86_64\", \"builddeps\": \"packit python3-build hatch python3-hatch-vcs\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}",
            "pkg_name": "packit",
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": "https://download.copr.fedorainfracloud.org/results/packit/packit-packit-2632/srpm-builds/09263162/packit-1.9.0.post1.dev7+g7ce12b95-1.20250710150535797465.pr2632.7.g7ce12b95.src.rpm",
            "uses_devel_repo": null,
            "sandbox": "packit/packit-packit-2632--packit",
            "results": {
                "name": "packit",
                "epoch": null,
                "version": "1.9.0.post1.dev7+g7ce12b95",
                "release": "1.20250710150535797465.pr2632.7.g7ce12b95",
                "exclusivearch": [],
                "excludearch": []
            },
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-42-x86_64/",
                    "id": "copr_packit_packit_stable",
                    "name": "Additional repo copr_packit_packit_stable"
                }
            ],
            "background": false,
            "destdir": "/var/lib/copr/public_html/results/packit/packit-packit-2632",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/packit-packit-2632",
            "result_dir": "09263162",
            "built_packages": "",
            "tags": [
                "arch_x86_64"
            ],
            "pkg_version": "1.9.0.post1.dev7+g7ce12b95-1.20250710150535797465.pr2632.7.g7ce12b95",
            "id": 9263162,
            "mockchain_macros": {
                "copr_username": "packit",
                "copr_projectname": "packit-packit-2632",
                "vendor": "Fedora Project COPR (packit/packit-packit-2632)"
            }
        }
    ]
}
[2025-07-10 15:05:40,824][  INFO][PID:3191111] Sending fedora-messaging bus message in build.end
[2025-07-10 15:05:40,861][  INFO][PID:3191111] Compressing /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/builder-live.log by gzip
[2025-07-10 15:05:40,864][  INFO][PID:3191111] Running command 'gzip /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/builder-live.log' as PID 3263353
[2025-07-10 15:05:40,881][  INFO][PID:3191111] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/builder-live.log)
[2025-07-10 15:05:40,882][  INFO][PID:3191111] Compressing /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/backend.log by gzip
[2025-07-10 15:05:40,885][  INFO][PID:3191111] Running command 'gzip /var/lib/copr/public_html/results/packit/packit-packit-2632/srpm-builds/09263162/backend.log' as PID 3263356