Linux
Windows
This document shows you how to resolve issues with the Compute Engine metadata
server.
Compute Engine VMs store metadata on a metadata server. VMs
automatically have access to the metadata server API without any additional
authorization. However, sometimes VMs might lose access to the metadata server
due to one of the following causes:
- Failure to resolve the metadata server domain name
- Connection to the metadata server is blocked by one of the following:
- OS level firewall configuration
- Proxy setup
- Custom routing
When VMs can't access the metadata server, some processes might fail.
For information about troubleshooting the
gke-metadata-server
, see
Troubleshoot GKE authentication issues
.
Before you begin
Common errors
The following are examples of that you might encounter if your VM fails to reach the metadata server:
curl: (6) Could not resolve host: metadata.google.internal
postAttribute error: Put "http://metadata.google.internal/computeMetadata/v1/instance/guest-attributes/guestInventory/ShortName": dial tcp: lookup metadata.google.internal on [::1]:53: read udp [::1]:58319->[::1]:53: read: connection refused
If your VM has lost access to the metadata server, do the following:
Linux
- Connect
to your Linux VM.
From your Linux VM, run the following commands to test connectivity to the metadata server:
Query the Domain Name Server:
nslookup metadata.google.internal
The output should look similar to the following:
Server: 169.254.169.254
Address: 169.254.169.254#53
Non-authoritative answer:
Name: metadata.google.internal
Address: 169.254.169.254
Check that the metadata server is reachable. To verify, run the
following commands:
ping -c 3 metadata.google.internal
The output should look similar to the following:
PING metadata.google.internal (169.254.169.254) 56(84) bytes of data.
64 bytes from metadata.google.internal (169.254.169.254): icmp_seq=1 ttl=255 time=0.812 ms
ping -c 3 169.254.169.254
The output should look similar to the following:
PING 169.254.169.254 (169.254.169.254) 56(84) bytes of data.
64 bytes from 169.254.169.254: icmp_seq=1 ttl=255 time=1.11 ms
If the output of the preceding commands matches the suggested
output, your VM is connected to the metadata server and no
further action is required. If the commands fail, do the
following:
Check that the name server is configured to the Metadata server:
cat /etc/resolv.conf
The output should look similar to the following:
domain ZONE.c.PROJECT_ID.internal
search ZONE.c.PROJECT_ID.internal. c.PROJECT_ID.internal. google.internal.
nameserver 169.254.169.254
If the output doesn't have the preceding lines, refer to your
operating system documentation for information about how to edit
the
DHCP Policy
to persist the name server configuration to
169.254.169.254
.
This is because changes to
/etc/resolv.conf
will be
overwritten in one hour if
zonal DNS
settings are applied to the VMs within your project. In case
your project is still using a
global DNS
, the
resolv.conf
file will be reverted to the default DHCP in 24
hours.
Check that the mapping between the metadata server domain name
and its IP address exist:
cat /etc/hosts
The following line should be in the output:
169.254.169.254 metadata.google.internal # Added by Google
If the output doesn't have the preceding line, run the following
command:
echo "169.254.169.254 metadata.google.internal" >> /etc/hosts
Windows
- Connect
to your Windows VM.
From your Windows VM, run the following commands:
Query the Domain Name Server:
nslookup metadata.google.internal
The output should look similar to the following:
Server: UnKnown
Address: 10.128.0.1
Non-authoritative answer:
Name: metadata.google.internal
Address: 169.254.169.254
Check that the metadata server is reachable. To verify, run the following commands:
ping -n 3 metadata.google.internal
The output should look similar to the following:
Pinging metadata.google.internal [169.254.169.254] with 32 bytes of data:
Reply from 169.254.169.254: bytes=32 time=1ms TTL=255
ping -n 3 169.254.169.254
The output should look similar to the following:
Pinging metadata.google.internal [169.254.169.254] with 32 bytes of data:
Reply from 169.254.169.254: bytes=32 time=1ms TTL=255
If the output of the preceding commands matches the suggested output,
your VM is connected to the metadata server and no further action
is required. If the commands fail, do the following:
Check that there's a persistent route to the metadata server by
running the command:
route print
The output should contain the following:
Persistent Routes:
Network Address Netmask Gateway Address Metric
169.254.169.254 255.255.255.255 On-link 1
If the output doesn't have the preceding line, add the
route using the following commands:
$Adapters = Get-NetKVMAdapterRegistry
$FirstAdapter = $Adapters | Select-Object -First 1
route /p add 169.254.169.254 mask 255.255.255.255 0.0.0.0 'if' $FirstAdapter.InterfaceIndex metric 1
Check that the mapping between the metadata server domain
name and its IP address exists:
type %WINDIR%\System32\Drivers\Etc\Hosts
The following line should be in the output:
169.254.169.254 metadata.google.internal # Added by Google
If the output doesn't have the preceding line, run the
following command:
echo 169.254.169.254 metadata.google.internal >> %WINDIR%\System32\Drivers\Etc\Hosts
Troubleshooting failed requests while using a network proxy
A network proxy server prevents VM's direct access to the Internet. All queries
sent from within a VM are handled by the proxy server instead.
When using an application that gets credentials from the metadata server, like
an authentication token, the VM requires direct access to the metadata server.
If the VM is behind a proxy, you must set the
NO_PROXY
configuration for both the IP address and Hostname.
If you don't set the
NO_PROXY
configuration, you might see errors
when running Google Cloud CLI commands or querying the metadata server directly
since the calls to
metadata.google.internal
will be sent to the
proxy without having them resolved locally on the instance itself.
The following is an example of an error that you might see:
ERROR 403 (Forbidden): Your client does not have permission to get URL
To resolve this proxy issue, add the metadata server hostname and IP address to
the environment variable
NO_PROXY
by doing the following:
Linux
- Connect
to your Linux VM.
From your Linux VM, run the following commands:
export no_proxy=169.254.169.254,metadata,metadata.google.internal
To save the changes, run the following command:
echo no_proxy=169.254.169.254,metadata,metadata.google.internal >> /etc/environment
Windows
- Connect
to your Windows VM.
From your Windows VM, run the following commands:
set NO_PROXY=169.254.169.254,metadata,metadata.google.internal
To save the changes, run the following command:
setx NO_PROXY 169.254.169.254,metadata,metadata.google.internal /m
The metadata server performs formatting checks to ensure that headers comply
with the header formatting guideline
RFC 7230 Section 3.2
.
If the header format fails these checks the following occurs:
The request is accepted. However, you'll receive recommendations that you
have VMs making requests to the metadata server with incorrectly formatted
headers. Recommendations are sent once per VM.
You can access these recommendations by using the
Google Cloud CLI, or the Recommender REST API.
After you have applied the recommendation,
set the recommendation state to
succeeded
.
Starting January 20, 2024, the metadata server denies any request with an
incorrectly formatted header.
The following shows examples of valid and invalid header request formats.
Not valid
: contains whitespace
between the header name and colon
Metadata-Flavor : Google
Valid
: no whitespace between header
name and colon, whitespace after the colon is ignored by the checker
Metadata-Flavor: Google
Valid
: no whitespace in header
Metadata-Flavor:Google
For more information about making queries to metadata server, see
Access VM metadata
.