한자로 한글한자자동변환기

GitHub Status - We are investigating reports of degraded performance.

We are investigating reports of degraded performance.

Incident Report for GitHub

Resolved

On May 14, 2024 between 18:00 UTC and 20:10 UTC, GitHub Actions performance was degraded and larger hosted runners using

linux-x64

images (Ubuntu20, Ubuntu22, Ubuntu24-beta) experienced longer than normal job start up times. Approximately 25% of all runs targeting larger hosted runners queued during this time were slow to start, with a median wait time of 1 minute, 55 seconds.

The issue was caused by a downstream dependency overloading, which impacted our machine setup process. Each larger hosted runner job is run on a fresh VM, and one of the setup steps is installing the Actions agent. Up to the point of this incident, we would pull the latest agent version from the GitHub release and install it on the VM. However, during this incident the speciific GitHub release for the Linux x64 Actions agent became overloaded, and our agent downloads were severely throttled. This throttling led to timeouts during the download, and caused our hosted runner system to conclude the VMs were failing to start. We need the Actions agent online to start serving jobs, and with the download timing out, our service assumed the runner wasn't starting up successfully. This failure to start up led to those VMs being reset again and again instead of serving jobs. We mitigated the issue by falling back to a cached version of the Actions agent present on our image.

We have further refined the fallback system to automatically use the cached agent binaries, and added new functionality to allow for easier agent downloading from other locations. Both of these measures should eliminate future impacts from similar downstream impact.

Posted May 14, 2024 - 21:04 UTC

Update

We are seeing recovery for queue times on Actions Larger Runners and are continuing to monitor full recovery.

Posted May 14, 2024 - 20:47 UTC

Update

We've applied a mitigation to fix the issues with queuing and running Actions jobs. We are seeing improvements in telemetry and are monitoring for full recovery.

Posted May 14, 2024 - 20:09 UTC

Update

We are continuing to investigate long queue times for Actions Larger Runners

Posted May 14, 2024 - 19:16 UTC

Update

We are investigating long queue times for Actions Larger Runners

Posted May 14, 2024 - 18:40 UTC

Investigating

We are currently investigating this issue.

Posted May 14, 2024 - 18:37 UTC

Product

Platform

Support

Company