[2025-07-23 18:13:59,823][  INFO][PID:1272330] Marking build as starting
[2025-07-23 18:13:59,851][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:13:59,852][  INFO][PID:1272330] VM allocation process starts
[2025-07-23 18:13:59,870][  INFO][PID:1272330] Trying to allocate VM: ResallocHost, ticket_id=4074008, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:14:04,872][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:14:09,875][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:14:14,878][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:14:17,961][  INFO][PID:1272330] Allocated host ResallocHost, ticket_id=4074008, hostname=2620:52:3:1:dead:beef:cafe:c1d6, name=copr_hv_x86_64_04_prod_05767467_20250723_181240, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:14:17,962][  INFO][PID:1272330] Allocating ssh connection to builder
[2025-07-23 18:14:17,963][  INFO][PID:1272330] Checking that builder machine is OK
[2025-07-23 18:14:18,500][  INFO][PID:1272330] Running remote command: copr-builder-ready epel-9-x86_64
[2025-07-23 18:24:18,830][  INFO][PID:1272330] Checking Red Hat subscription (attempt #1)
Checking Red Hat subscription (attempt #2)
Checking Red Hat subscription (attempt #3)
Checking Red Hat subscription (attempt #4)
Checking Red Hat subscription (attempt #5)
Checking Red Hat subscription (attempt #6)
Checking Red Hat subscription (attempt #7)
Checking Red Hat subscription (attempt #8)
Checking Red Hat subscription (attempt #9)
Checking Red Hat subscription (attempt #10)
Checking Red Hat subscription (attempt #11)
Checking Red Hat subscription (attempt #12)
Checking Red Hat subscription (attempt #13)
Checking Red Hat subscription (attempt #14)
Checking Red Hat subscription (attempt #15)
Checking Red Hat subscription (attempt #16)
Checking Red Hat subscription (attempt #17)
Checking Red Hat subscription (attempt #18)
Checking Red Hat subscription (attempt #19)
Checking Red Hat subscription (attempt #20)
Checking Red Hat subscription (attempt #21)
Waiting for Red Hat subscription timeouted!

[2025-07-23 18:24:18,831][  INFO][PID:1272330] Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c1d6' (ED25519) to the list of known hosts.

[2025-07-23 18:24:18,831][ ERROR][PID:1272330] Re-try request for task on 'ResallocHost, ticket_id=4074008, hostname=2620:52:3:1:dead:beef:cafe:c1d6, name=copr_hv_x86_64_04_prod_05767467_20250723_181240, requested_tags=['copr_builder', 'arch_x86_64']': Builder wasn't ready, trying a new one
[2025-07-23 18:24:18,832][  INFO][PID:1272330] Releasing VM back to pool
[2025-07-23 18:24:18,843][  INFO][PID:1272330] Retry #1 (on other host)
[2025-07-23 18:24:18,844][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:24:18,844][  INFO][PID:1272330] VM allocation process starts
[2025-07-23 18:24:18,855][  INFO][PID:1272330] Trying to allocate VM: ResallocHost, ticket_id=4074216, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:24:21,871][  INFO][PID:1272330] Allocated host ResallocHost, ticket_id=4074216, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:24:21,872][  INFO][PID:1272330] Allocating ssh connection to builder
[2025-07-23 18:24:21,872][  INFO][PID:1272330] Checking that builder machine is OK
[2025-07-23 18:24:22,331][  INFO][PID:1272330] Running remote command: copr-builder-ready epel-9-x86_64
[2025-07-23 18:34:22,528][  INFO][PID:1272330] Checking Red Hat subscription (attempt #1)
Checking Red Hat subscription (attempt #2)
Checking Red Hat subscription (attempt #3)
Checking Red Hat subscription (attempt #4)
Checking Red Hat subscription (attempt #5)
Checking Red Hat subscription (attempt #6)
Checking Red Hat subscription (attempt #7)
Checking Red Hat subscription (attempt #8)
Checking Red Hat subscription (attempt #9)
Checking Red Hat subscription (attempt #10)
Checking Red Hat subscription (attempt #11)
Checking Red Hat subscription (attempt #12)
Checking Red Hat subscription (attempt #13)
Checking Red Hat subscription (attempt #14)
Checking Red Hat subscription (attempt #15)
Checking Red Hat subscription (attempt #16)
Checking Red Hat subscription (attempt #17)
Checking Red Hat subscription (attempt #18)
Checking Red Hat subscription (attempt #19)
Checking Red Hat subscription (attempt #20)
Checking Red Hat subscription (attempt #21)
Waiting for Red Hat subscription timeouted!

[2025-07-23 18:34:22,529][  INFO][PID:1272330] Warning: Permanently added '13.220.248.225' (ED25519) to the list of known hosts.

[2025-07-23 18:34:22,530][ ERROR][PID:1272330] Re-try request for task on 'ResallocHost, ticket_id=4074216, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']': Builder wasn't ready, trying a new one
[2025-07-23 18:34:22,530][  INFO][PID:1272330] Releasing VM back to pool
[2025-07-23 18:34:22,544][  INFO][PID:1272330] Retry #2 (on other host)
[2025-07-23 18:34:22,545][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:34:22,546][  INFO][PID:1272330] VM allocation process starts
[2025-07-23 18:34:22,559][  INFO][PID:1272330] Trying to allocate VM: ResallocHost, ticket_id=4074450, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:34:27,560][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:34:32,563][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:34:34,602][  INFO][PID:1272330] Allocated host ResallocHost, ticket_id=4074450, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:34:34,602][  INFO][PID:1272330] Allocating ssh connection to builder
[2025-07-23 18:34:34,603][  INFO][PID:1272330] Checking that builder machine is OK
[2025-07-23 18:34:35,028][  INFO][PID:1272330] Running remote command: copr-builder-ready epel-9-x86_64
[2025-07-23 18:44:35,227][  INFO][PID:1272330] Checking Red Hat subscription (attempt #1)
Checking Red Hat subscription (attempt #2)
Checking Red Hat subscription (attempt #3)
Checking Red Hat subscription (attempt #4)
Checking Red Hat subscription (attempt #5)
Checking Red Hat subscription (attempt #6)
Checking Red Hat subscription (attempt #7)
Checking Red Hat subscription (attempt #8)
Checking Red Hat subscription (attempt #9)
Checking Red Hat subscription (attempt #10)
Checking Red Hat subscription (attempt #11)
Checking Red Hat subscription (attempt #12)
Checking Red Hat subscription (attempt #13)
Checking Red Hat subscription (attempt #14)
Checking Red Hat subscription (attempt #15)
Checking Red Hat subscription (attempt #16)
Checking Red Hat subscription (attempt #17)
Checking Red Hat subscription (attempt #18)
Checking Red Hat subscription (attempt #19)
Checking Red Hat subscription (attempt #20)
Checking Red Hat subscription (attempt #21)
Waiting for Red Hat subscription timeouted!

[2025-07-23 18:44:35,228][  INFO][PID:1272330] Warning: Permanently added '13.220.248.225' (ED25519) to the list of known hosts.

[2025-07-23 18:44:35,230][ ERROR][PID:1272330] Re-try request for task on 'ResallocHost, ticket_id=4074450, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']': Builder wasn't ready, trying a new one
[2025-07-23 18:44:35,231][  INFO][PID:1272330] Releasing VM back to pool
[2025-07-23 18:44:35,268][  INFO][PID:1272330] Retry #3 (on other host)
[2025-07-23 18:44:35,271][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:44:35,274][  INFO][PID:1272330] VM allocation process starts
[2025-07-23 18:44:35,294][  INFO][PID:1272330] Trying to allocate VM: ResallocHost, ticket_id=4074735, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:44:40,298][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:44:45,301][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:44:47,360][  INFO][PID:1272330] Allocated host ResallocHost, ticket_id=4074735, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:44:47,363][  INFO][PID:1272330] Allocating ssh connection to builder
[2025-07-23 18:44:47,365][  INFO][PID:1272330] Checking that builder machine is OK
[2025-07-23 18:44:47,753][  INFO][PID:1272330] Running remote command: copr-builder-ready epel-9-x86_64
[2025-07-23 18:54:47,968][  INFO][PID:1272330] Checking Red Hat subscription (attempt #1)
Checking Red Hat subscription (attempt #2)
Checking Red Hat subscription (attempt #3)
Checking Red Hat subscription (attempt #4)
Checking Red Hat subscription (attempt #5)
Checking Red Hat subscription (attempt #6)
Checking Red Hat subscription (attempt #7)
Checking Red Hat subscription (attempt #8)
Checking Red Hat subscription (attempt #9)
Checking Red Hat subscription (attempt #10)
Checking Red Hat subscription (attempt #11)
Checking Red Hat subscription (attempt #12)
Checking Red Hat subscription (attempt #13)
Checking Red Hat subscription (attempt #14)
Checking Red Hat subscription (attempt #15)
Checking Red Hat subscription (attempt #16)
Checking Red Hat subscription (attempt #17)
Checking Red Hat subscription (attempt #18)
Checking Red Hat subscription (attempt #19)
Checking Red Hat subscription (attempt #20)
Checking Red Hat subscription (attempt #21)
Waiting for Red Hat subscription timeouted!

[2025-07-23 18:54:47,968][  INFO][PID:1272330] Warning: Permanently added '13.220.248.225' (ED25519) to the list of known hosts.

[2025-07-23 18:54:47,969][ ERROR][PID:1272330] Re-try request for task on 'ResallocHost, ticket_id=4074735, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']': Builder wasn't ready, trying a new one
[2025-07-23 18:54:47,969][  INFO][PID:1272330] Releasing VM back to pool
[2025-07-23 18:54:48,010][  INFO][PID:1272330] Retry #4 (on other host)
[2025-07-23 18:54:48,011][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:54:48,012][  INFO][PID:1272330] VM allocation process starts
[2025-07-23 18:54:48,025][  INFO][PID:1272330] Trying to allocate VM: ResallocHost, ticket_id=4074941, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:54:53,026][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 18:54:54,050][  INFO][PID:1272330] Allocated host ResallocHost, ticket_id=4074941, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 18:54:54,050][  INFO][PID:1272330] Allocating ssh connection to builder
[2025-07-23 18:54:54,051][  INFO][PID:1272330] Checking that builder machine is OK
[2025-07-23 18:54:54,342][  INFO][PID:1272330] Running remote command: copr-builder-ready epel-9-x86_64
[2025-07-23 19:04:54,571][  INFO][PID:1272330] Checking Red Hat subscription (attempt #1)
Checking Red Hat subscription (attempt #2)
Checking Red Hat subscription (attempt #3)
Checking Red Hat subscription (attempt #4)
Checking Red Hat subscription (attempt #5)
Checking Red Hat subscription (attempt #6)
Checking Red Hat subscription (attempt #7)
Checking Red Hat subscription (attempt #8)
Checking Red Hat subscription (attempt #9)
Checking Red Hat subscription (attempt #10)
Checking Red Hat subscription (attempt #11)
Checking Red Hat subscription (attempt #12)
Checking Red Hat subscription (attempt #13)
Checking Red Hat subscription (attempt #14)
Checking Red Hat subscription (attempt #15)
Checking Red Hat subscription (attempt #16)
Checking Red Hat subscription (attempt #17)
Checking Red Hat subscription (attempt #18)
Checking Red Hat subscription (attempt #19)
Checking Red Hat subscription (attempt #20)
Checking Red Hat subscription (attempt #21)
Waiting for Red Hat subscription timeouted!

[2025-07-23 19:04:54,577][  INFO][PID:1272330] Warning: Permanently added '13.220.248.225' (ED25519) to the list of known hosts.

[2025-07-23 19:04:54,578][ ERROR][PID:1272330] Re-try request for task on 'ResallocHost, ticket_id=4074941, hostname=13.220.248.225, name=aws_x86_64_normalreserved_prod_05767676_20250723_182219, requested_tags=['copr_builder', 'arch_x86_64']': Builder wasn't ready, trying a new one
[2025-07-23 19:04:54,579][  INFO][PID:1272330] Releasing VM back to pool
[2025-07-23 19:04:54,611][  INFO][PID:1272330] Retry #5 (on other host)
[2025-07-23 19:04:54,614][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:04:54,619][  INFO][PID:1272330] VM allocation process starts
[2025-07-23 19:04:54,666][  INFO][PID:1272330] Trying to allocate VM: ResallocHost, ticket_id=4075144, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 19:04:59,669][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:04,672][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:06,764][  INFO][PID:1272330] Allocated host ResallocHost, ticket_id=4075144, hostname=107.20.22.185, name=aws_x86_64_normalreserved_prod_05768611_20250723_190258, requested_tags=['copr_builder', 'arch_x86_64']
[2025-07-23 19:05:06,767][  INFO][PID:1272330] Allocating ssh connection to builder
[2025-07-23 19:05:06,768][  INFO][PID:1272330] Checking that builder machine is OK
[2025-07-23 19:05:07,181][  INFO][PID:1272330] Running remote command: copr-builder-ready epel-9-x86_64
[2025-07-23 19:05:07,453][  INFO][PID:1272330] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-07-23 19:05:07,456][  INFO][PID:1272330] Filling build.info file with builder info
[2025-07-23 19:05:07,458][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:07,459][  INFO][PID:1272330] Not re-notifying FE and msg buses for the new host.
[2025-07-23 19:05:07,461][  INFO][PID:1272330] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9314872-epel-9-x86_64 --chroot epel-9-x86_64 --detached
[2025-07-23 19:05:07,899][  INFO][PID:1272330] Downloading the builder-live.log file, attempt 1
[2025-07-23 19:05:07,907][  INFO][PID:1272330] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.20.22.185 copr-rpmbuild-log
[2025-07-23 19:05:12,920][  INFO][PID:1272330] Periodic builder liveness probe: alive
[2025-07-23 19:05:12,921][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:17,925][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:22,928][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:27,933][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:32,955][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:37,968][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:42,977][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:47,978][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:52,985][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:05:57,998][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:03,009][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:08,013][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:13,019][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:18,021][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:23,027][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:28,035][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:33,039][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:38,046][  INFO][PID:1272330] Checking for cancel request
[2025-07-23 19:06:41,249][  INFO][PID:1272330] Downloading results from builder
[2025-07-23 19:06:41,252][  INFO][PID:1272330] rsyncing of mockbuilder@107.20.22.185:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot started
[2025-07-23 19:06:41,260][  INFO][PID:1272330] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@107.20.22.185:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/ &> /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/build-09314872.rsync.log
[2025-07-23 19:06:41,782][  INFO][PID:1272330] rsyncing finished.
[2025-07-23 19:06:41,784][  INFO][PID:1272330] Releasing VM back to pool
[2025-07-23 19:06:41,857][  INFO][PID:1272330] Searching for 'success' file in resultdir
[2025-07-23 19:06:41,861][ ERROR][PID:1272330] Build failed: Backend process error: No success file => build failure
[2025-07-23 19:06:41,866][  INFO][PID:1272330] Finished build: id=9314872 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/matthewdva/certbot chroot=epel-9-x86_64 
[2025-07-23 19:06:41,882][ ERROR][PID:1272330] Backend process error: No success file => build failure
[2025-07-23 19:06:41,884][  INFO][PID:1272330] Worker failed build, took 94.42436933517456
[2025-07-23 19:06:41,886][  INFO][PID:1272330] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "matthewdva",
            "project_name": "certbot",
            "project_dirname": "certbot",
            "submitter": "https://src.fedoraproject.org/user/releng",
            "ended_on": 1753297601.8841434,
            "started_on": 1753297507.459774,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-9-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9314872-epel-9-x86_64",
            "build_id": 9314872,
            "package_name": "certbot",
            "package_version": "4.1.1-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/matthewdva/certbot/certbot",
            "git_hash": "5e566e223ae92657f3d403b7eaabe2f19e6d28f2",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "matthewdva/certbot--https://src.fedoraproject.org/user/releng",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/matthewdva/certbot/epel-9-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://dl.fedoraproject.org/pub/epel/testing/$releasever/Everything/$basearch",
                    "id": "https_dl_fedoraproject_org_pub_epel_testing_releasever_Everything_basearch",
                    "name": "Additional repo https_dl_fedoraproject_org_pub_epel_testing_releasever_Everything_basearch"
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/matthewdva/certbot",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/matthewdva/certbot",
            "result_dir": "09314872-certbot",
            "built_packages": "",
            "id": 9314872,
            "mockchain_macros": {
                "copr_username": "matthewdva",
                "copr_projectname": "certbot",
                "vendor": "Fedora Project COPR (matthewdva/certbot)"
            }
        }
    ]
}
[2025-07-23 19:06:41,927][  INFO][PID:1272330] Sending fedora-messaging bus message in build.end
[2025-07-23 19:06:42,995][  INFO][PID:1272330] Compressing /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/builder-live.log by gzip
[2025-07-23 19:06:42,999][  INFO][PID:1272330] Running command 'gzip /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/builder-live.log' as PID 1724724
[2025-07-23 19:06:43,030][  INFO][PID:1272330] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/builder-live.log)
[2025-07-23 19:06:43,040][  INFO][PID:1272330] Compressing /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/backend.log by gzip
[2025-07-23 19:06:43,042][  INFO][PID:1272330] Running command 'gzip /var/lib/copr/public_html/results/matthewdva/certbot/epel-9-x86_64/09314872-certbot/backend.log' as PID 1724728