한자로 한글한자자동변환기

GitHub Status - Incident with Codespaces and API Requests

Incident with Codespaces and API Requests

Incident Report for GitHub

Resolved

On March 15, 2024, between 19:42 UTC and 20:24 UTC several services were degraded due to a regression in calling the permissions system.

New GitHub Codespaces could not be created, as were Codespaces sessions that required minting a new auth token.

Actions saw delays and infrastructure failures due to the upstream dependency on fetching tokens for the repository for runs to successfully execute.

GitHub Pages were affected due to the impact on Actions, resulting in 1266 page builds failing, which at the low point represented 33% of page builds failing. This resulted in page edits not being reflected on those impacted sites.

We deployed an application update that included a newer version of our database query builder. The new version uses a newer MySQL syntax for upsert queries that is not supported by the database proxy service we use for some of our production-environment database clusters. This incompatibility impacted the permissions cluster specifically, causing requests that attempted such queries to fail.

We responded by rolling back the deployment, restoring the previous query use, and thus mitigated the incident.

We have identified and corrected a misconfiguration of the permissions cluster in our development and CI environments that will ensure queries utilize the proxy service to prevent future syntax additions causing issues in production.

Posted Mar 15, 2024 - 20:24 UTC

Update

Codespaces is operating normally.

Posted Mar 15, 2024 - 20:21 UTC

Update

API Requests is operating normally.

Posted Mar 15, 2024 - 20:20 UTC

Update

We rolled back the most recent deployment and are seeing improvements across all services, and will continue to monitor for additional impact.

Posted Mar 15, 2024 - 20:17 UTC

Update

API Requests is experiencing degraded performance. We are continuing to investigate.

Posted Mar 15, 2024 - 20:11 UTC

Update

Codespaces is experiencing degraded availability. We are continuing to investigate.

Posted Mar 15, 2024 - 20:03 UTC

Update

API Requests is experiencing degraded availability. We are continuing to investigate.

Posted Mar 15, 2024 - 20:03 UTC

Update

API Requests is experiencing degraded performance. We are continuing to investigate.

Posted Mar 15, 2024 - 20:00 UTC

Investigating

We are investigating reports of degraded performance for Codespaces

Posted Mar 15, 2024 - 19:55 UTC

This incident affected: API Requests and Codespaces.

Product

Platform

Support

Company