[2025-05-14 16:31:43,740][  INFO][PID:3078278] Marking build as starting
[2025-05-14 16:31:43,812][  INFO][PID:3078278] Trying to allocate VM: ResallocHost, ticket_id=2991391, requested_tags=['copr_builder', 'arch_x86_64']
[2025-05-14 16:31:46,876][  INFO][PID:3078278] Allocated host ResallocHost, ticket_id=2991391, hostname=34.205.19.130, name=aws_x86_64_spot_prod_03946432_20250514_161820, requested_tags=['copr_builder', 'arch_x86_64']
[2025-05-14 16:31:46,881][  INFO][PID:3078278] Allocating ssh connection to builder
[2025-05-14 16:31:46,884][  INFO][PID:3078278] Checking that builder machine is OK
[2025-05-14 16:31:49,055][  INFO][PID:3078278] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-05-14 16:31:49,276][  INFO][PID:3078278] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-05-14 16:31:49,278][  INFO][PID:3078278] Filling build.info file with builder info
[2025-05-14 16:31:49,281][  INFO][PID:3078278] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 180000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@python",
            "project_name": "python3.14",
            "project_dirname": "python3.14",
            "submitter": "churchyard",
            "ended_on": null,
            "started_on": 1747240309.2812524,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9039072-fedora-rawhide-x86_64",
            "build_id": 9039072,
            "package_name": "xpra",
            "package_version": "5.0.10-5",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python3.14/xpra",
            "git_hash": "30f565421954a3e9d8008d114e0b5593a1c84118",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@python/python3.14--churchyard",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 9,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python3.14/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 9
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@python/python3.14",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14",
            "result_dir": "09039072-xpra",
            "built_packages": "",
            "id": 9039072,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python3.14",
                "vendor": "Fedora Project COPR (@python/python3.14)"
            }
        }
    ]
}
[2025-05-14 16:31:49,409][  INFO][PID:3078278] Sending fedora-messaging bus message in build.start
[2025-05-14 16:31:50,056][  INFO][PID:3078278] Sending fedora-messaging bus message in chroot.start
[2025-05-14 16:31:50,077][  INFO][PID:3078278] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9039072-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-05-14 16:31:50,483][  INFO][PID:3078278] Downloading the builder-live.log file, attempt 1
[2025-05-14 16:31:50,486][  INFO][PID:3078278] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.19.130 copr-rpmbuild-log
[2025-05-14 16:31:55,514][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 16:34:46,402][ ERROR][PID:3078278] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-05-14 16:34:56,405][WARNING][PID:3078278] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-05-14 16:35:06,408][  INFO][PID:3078278] Downloading the builder-live.log file, attempt 2
[2025-05-14 16:35:06,416][  INFO][PID:3078278] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.19.130 copr-rpmbuild-log
[2025-05-14 16:35:06,427][ ERROR][PID:3078278] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-05-14 16:35:16,430][WARNING][PID:3078278] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-05-14 16:35:26,432][  INFO][PID:3078278] Downloading the builder-live.log file, attempt 3
[2025-05-14 16:35:26,435][  INFO][PID:3078278] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.19.130 copr-rpmbuild-log
[2025-05-14 16:35:26,448][ ERROR][PID:3078278] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-05-14 16:35:36,482][WARNING][PID:3078278] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-05-14 16:35:46,484][  INFO][PID:3078278] Downloading the builder-live.log file, attempt 4
[2025-05-14 16:35:46,487][  INFO][PID:3078278] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.205.19.130 copr-rpmbuild-log
[2025-05-14 16:36:46,568][ ERROR][PID:3078278] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-05-14 16:36:56,575][ ERROR][PID:3078278] Giving up for unstable SSH, failures: 1747240496.4058232, 1747240516.4300835, 1747240536.4823556, 1747240616.5755434
[2025-05-14 16:36:56,578][ ERROR][PID:3078278] Re-try request for task on 'ResallocHost, ticket_id=2991391, hostname=34.205.19.130, name=aws_x86_64_spot_prod_03946432_20250514_161820, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-05-14 16:36:56,579][  INFO][PID:3078278] Releasing VM back to pool
[2025-05-14 16:36:56,596][  INFO][PID:3078278] Retry #1 (on other host)
[2025-05-14 16:36:56,624][  INFO][PID:3078278] Trying to allocate VM: ResallocHost, ticket_id=2991556, requested_tags=['copr_builder', 'arch_x86_64']
[2025-05-14 16:36:59,759][  INFO][PID:3078278] Allocated host ResallocHost, ticket_id=2991556, hostname=52.87.170.46, name=aws_x86_64_normalreserved_prod_03946865_20250514_162600, requested_tags=['copr_builder', 'arch_x86_64']
[2025-05-14 16:36:59,764][  INFO][PID:3078278] Allocating ssh connection to builder
[2025-05-14 16:36:59,767][  INFO][PID:3078278] Checking that builder machine is OK
[2025-05-14 16:37:00,289][  INFO][PID:3078278] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-05-14 16:37:00,499][  INFO][PID:3078278] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-05-14 16:37:00,501][  INFO][PID:3078278] Filling build.info file with builder info
[2025-05-14 16:37:00,511][  INFO][PID:3078278] Not re-notifying FE and msg buses for the new host.
[2025-05-14 16:37:00,514][  INFO][PID:3078278] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9039072-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-05-14 16:37:00,889][  INFO][PID:3078278] Downloading the builder-live.log file, attempt 1
[2025-05-14 16:37:00,893][  INFO][PID:3078278] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.87.170.46 copr-rpmbuild-log
[2025-05-14 16:41:56,197][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 16:51:56,751][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 17:01:57,185][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 17:11:57,709][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 17:21:58,475][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 17:31:58,912][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 17:41:59,212][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 17:51:59,574][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 18:01:59,909][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 18:12:00,514][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 18:22:01,360][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 18:32:02,332][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 18:42:03,412][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 18:52:04,096][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 19:02:04,550][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 19:12:05,243][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 19:22:05,852][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 19:32:06,614][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 19:42:07,488][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 19:52:07,979][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 20:02:08,289][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 20:12:08,684][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 20:22:09,251][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 20:32:09,489][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 20:42:09,753][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 20:52:09,974][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 21:02:10,363][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 21:12:11,134][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 21:22:12,057][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 21:32:12,477][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 21:42:12,778][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 21:52:12,964][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 22:02:13,202][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 22:12:13,623][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 22:22:14,081][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 22:32:14,276][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 22:42:14,462][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 22:52:14,671][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 23:02:14,874][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 23:12:15,251][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 23:22:15,468][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 23:32:15,626][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 23:42:15,769][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-14 23:52:15,938][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 00:02:16,091][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 00:12:16,398][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 00:22:16,633][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 00:32:16,793][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 00:42:17,011][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 00:52:17,188][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 01:02:17,361][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 01:12:17,606][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 01:22:18,255][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 01:32:18,501][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 01:42:18,680][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 01:52:18,853][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 02:02:19,152][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 02:12:19,536][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 02:22:19,848][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 02:32:20,167][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 02:42:20,347][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 02:52:20,510][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 03:02:20,687][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 03:12:20,947][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 03:22:21,574][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 03:32:21,954][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 03:42:22,288][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 03:52:22,522][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 04:02:22,763][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 04:12:23,611][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 04:22:24,934][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 04:32:25,875][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 04:42:26,480][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 04:52:26,946][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 05:02:27,615][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 05:12:28,346][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 05:22:29,357][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 05:32:30,707][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 05:42:31,654][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 05:52:32,053][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 06:02:32,321][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 06:12:33,235][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 06:22:33,823][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 06:32:34,243][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 06:42:34,706][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 06:52:35,296][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 07:02:36,063][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 07:12:36,805][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 07:22:37,212][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 07:32:37,528][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 07:42:37,814][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 07:52:38,331][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 08:02:39,284][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 08:12:40,313][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 08:22:41,729][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 08:32:42,765][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 08:42:43,535][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 08:52:45,012][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 09:02:47,968][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 09:12:50,036][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 09:22:52,770][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 09:32:54,857][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 09:42:57,185][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 09:52:59,537][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 10:03:01,988][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 10:13:03,601][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 10:23:05,280][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 10:33:06,685][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 10:43:09,175][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 10:53:11,071][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 11:03:11,885][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 11:13:13,120][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 11:23:15,013][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 11:33:16,108][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 11:43:17,224][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 11:53:18,339][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 12:03:19,276][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 12:13:20,261][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 12:23:21,370][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 12:33:22,167][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 12:43:22,817][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 12:53:23,662][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 13:03:24,396][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 13:13:25,315][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 13:23:25,962][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 13:33:26,680][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 13:43:27,183][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 13:53:27,721][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 14:03:28,340][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 14:13:28,839][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 14:23:29,349][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 14:33:29,923][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 14:43:30,449][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 14:53:30,942][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 15:03:31,526][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 15:13:32,227][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 15:23:32,775][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 15:33:33,136][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 15:43:33,650][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 15:53:34,042][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 16:03:34,547][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 16:13:35,068][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 16:23:35,652][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 16:33:36,271][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 16:43:36,792][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 16:53:37,309][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 17:03:37,910][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 17:13:38,765][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 17:23:39,596][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 17:33:40,549][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 17:43:41,217][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 17:53:41,942][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 18:03:42,458][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 18:13:43,015][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 18:23:43,888][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 18:33:44,913][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 18:43:45,730][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 18:53:46,722][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 19:03:47,491][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 19:13:48,312][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 19:23:49,268][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 19:33:49,983][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 19:43:50,479][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 19:53:50,792][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 20:03:51,010][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 20:13:51,576][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 20:23:51,880][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 20:33:52,149][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 20:43:52,331][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 20:53:52,517][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 21:03:52,708][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 21:13:53,450][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 21:23:53,960][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 21:33:54,195][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 21:43:54,377][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 21:53:54,545][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 22:03:54,714][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 22:13:55,015][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 22:23:55,199][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 22:33:55,368][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 22:43:55,537][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 22:53:55,718][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 23:03:55,872][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 23:13:56,158][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 23:23:56,360][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 23:33:56,605][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 23:43:56,842][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-15 23:53:57,011][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 00:03:57,182][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 00:13:57,406][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 00:23:57,581][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 00:33:57,769][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 00:43:57,949][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 00:53:58,164][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 01:03:58,379][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 01:13:58,589][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 01:23:58,982][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 01:33:59,181][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 01:43:59,359][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 01:53:59,550][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 02:03:59,850][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 02:14:00,455][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 02:24:01,044][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 02:34:01,447][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 02:44:01,681][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 02:54:01,874][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 03:04:02,031][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 03:14:02,258][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 03:24:02,451][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 03:34:02,682][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 03:44:02,913][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 03:54:03,378][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 04:04:03,557][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 04:14:04,186][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 04:24:04,715][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 04:34:05,044][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 04:44:05,260][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 04:54:05,462][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 05:04:05,655][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 05:14:05,949][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 05:24:06,108][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 05:34:06,283][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 05:44:07,032][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 05:54:07,238][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 06:04:07,434][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 06:14:07,680][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 06:24:07,887][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 06:34:08,085][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 06:44:08,282][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 06:54:08,438][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 07:04:08,601][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 07:14:08,838][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 07:24:09,007][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 07:34:09,192][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 07:44:09,453][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 07:54:09,647][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 08:04:09,859][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 08:14:10,276][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 08:24:10,514][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 08:34:10,698][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 08:44:10,856][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 08:54:11,030][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 09:04:11,228][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 09:14:11,514][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 09:24:11,708][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 09:34:11,905][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 09:44:12,057][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 09:54:12,260][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 10:04:12,447][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 10:14:12,787][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 10:24:13,115][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 10:34:13,294][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 10:44:13,456][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 10:54:13,640][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 11:04:13,829][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 11:14:14,148][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 11:24:14,311][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 11:34:14,503][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 11:44:14,740][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 11:54:14,955][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 12:04:15,156][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 12:14:15,398][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 12:24:15,568][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 12:34:15,726][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 12:44:15,927][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 12:54:16,097][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 13:04:16,305][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 13:14:16,575][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 13:24:16,766][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 13:34:16,955][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 13:44:17,173][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 13:54:17,339][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 14:04:17,527][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 14:14:18,007][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 14:24:18,373][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 14:34:18,543][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 14:44:18,767][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 14:54:18,997][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 15:04:19,203][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 15:14:19,526][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 15:24:19,829][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 15:34:20,018][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 15:44:20,202][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 15:54:20,373][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 16:04:20,580][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 16:14:20,861][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 16:24:21,162][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 16:34:21,483][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 16:44:21,738][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 16:54:21,919][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 17:04:22,234][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 17:14:22,922][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 17:24:23,422][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 17:34:23,627][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 17:44:23,807][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 17:54:24,038][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 18:04:24,245][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 18:14:24,532][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 18:24:24,704][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 18:34:25,056][  INFO][PID:3078278] Periodic builder liveness probe: alive
[2025-05-16 18:37:03,700][  INFO][PID:3078278] Downloading results from builder
[2025-05-16 18:37:03,731][  INFO][PID:3078278] rsyncing of mockbuilder@52.87.170.46:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra started
[2025-05-16 18:37:03,765][  INFO][PID:3078278] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@52.87.170.46:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/ &> /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/build-09039072.rsync.log
[2025-05-16 18:37:04,557][  INFO][PID:3078278] rsyncing finished.
[2025-05-16 18:37:04,571][  INFO][PID:3078278] Releasing VM back to pool
[2025-05-16 18:37:04,621][  INFO][PID:3078278] Searching for 'success' file in resultdir
[2025-05-16 18:37:04,629][ ERROR][PID:3078278] Build failed: Backend process error: No success file => build failure
[2025-05-16 18:37:04,728][  INFO][PID:3078278] Finished build: id=9039072 failed=True timeout=180000 destdir=/var/lib/copr/public_html/results/@python/python3.14 chroot=fedora-rawhide-x86_64 
[2025-05-16 18:37:04,873][ ERROR][PID:3078278] Backend process error: No success file => build failure
[2025-05-16 18:37:04,885][  INFO][PID:3078278] Worker failed build, took 180004.3721807003
[2025-05-16 18:37:04,910][  INFO][PID:3078278] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 180000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@python",
            "project_name": "python3.14",
            "project_dirname": "python3.14",
            "submitter": "churchyard",
            "ended_on": 1747420624.8840694,
            "started_on": 1747240620.5118887,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9039072-fedora-rawhide-x86_64",
            "build_id": 9039072,
            "package_name": "xpra",
            "package_version": "5.0.10-5",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@python/python3.14/xpra",
            "git_hash": "30f565421954a3e9d8008d114e0b5593a1c84118",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@python/python3.14--churchyard",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 9,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@python/python3.14/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 9
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@python/python3.14",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@python/python3.14",
            "result_dir": "09039072-xpra",
            "built_packages": "",
            "id": 9039072,
            "mockchain_macros": {
                "copr_username": "@python",
                "copr_projectname": "python3.14",
                "vendor": "Fedora Project COPR (@python/python3.14)"
            }
        }
    ]
}
[2025-05-16 18:37:05,767][  INFO][PID:3078278] Sending fedora-messaging bus message in build.end
[2025-05-16 18:37:06,244][  INFO][PID:3078278] Compressing /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/builder-live.log by gzip
[2025-05-16 18:37:06,309][  INFO][PID:3078278] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/builder-live.log' as PID 223914
[2025-05-16 18:38:20,618][  INFO][PID:3078278] Finished after 74 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/builder-live.log)
[2025-05-16 18:38:20,629][  INFO][PID:3078278] Compressing /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/backend.log by gzip
[2025-05-16 18:38:20,646][  INFO][PID:3078278] Running command 'gzip /var/lib/copr/public_html/results/@python/python3.14/fedora-rawhide-x86_64/09039072-xpra/backend.log' as PID 236893