In our previous article, we implemented a mechanism that reads the browser’s language settings via navigator.languages and automatically redirects visitors from the top page to /en/ or /zh/ when their language isn’t Japanese, and explained how it worked. We have now removed that mechanism. The trigger was a report from Google Search Console.

Observation

In Google Search Console’s Page Indexing report, our English top page (https://kobayaxi.com/en/) had the following status.

Duplicate, Google chose different canonical than user

The same help page explains the condition under which this status appears:

This page is marked as canonical for a set of pages, but Google thinks another URL makes a better canonical.

The HTML for /en/ declares itself as canonical via <link rel="canonical" href="https://kobayaxi.com/en/">, so this status means Google adopted a different URL (most likely the Japanese top page, /) as canonical instead.

We first suspected a mistake in the canonical or hreflang tag implementation itself. Checking the HTML actually being served, we found the following.

Item/ (Japanese)/en/ (English)
canonicalhttps://kobayaxi.com/https://kobayaxi.com/en/
hreflang=jahttps://kobayaxi.com/https://kobayaxi.com/
hreflang=enhttps://kobayaxi.com/en/https://kobayaxi.com/en/
hreflang=zh-CNhttps://kobayaxi.com/zh/https://kobayaxi.com/zh/
hreflang=x-defaulthttps://kobayaxi.com/https://kobayaxi.com/

Each page correctly self-references as canonical, and the hreflang tags are reciprocally consistent. We ruled out a tagging mistake as the cause, and turned to the automatic redirect implemented on the top page.

Cause and Mechanism

Google Search Central’s “How Google crawls locale-adaptive pages” describes Googlebot’s crawling behavior as follows.

Googlebot crawls with IP addresses based outside the USA, in addition to the US-based IP addresses…the crawler sends HTTP requests without setting Accept-Language in the request header.

The redirect logic we had in place read navigator.languages (the browser’s ordered list of preferred languages, normally derived from the Accept-Language setting), and ran location.replace() to /en/ or /zh/ whenever the first entry didn’t start with "ja". But as stated above, Googlebot does not send an Accept-Language header with its requests. If navigator.languages falls back to some default of the rendering environment when that setting is absent, then which branch this script takes when Googlebot executes it is neither controlled nor predictable from our side.

Google explicitly warns against this kind of automatic redirect in “Localized versions of your pages.”

Avoid automatically redirecting users from one language version of a site to a different language version of a site. For example, don’t redirect based on what you think the user’s language may be. These redirections could prevent users (and search engines) from viewing all the versions of your site.

We were aware of this guidance when we wrote the previous article, and had implemented three safeguards accordingly:

  • Limit the automatic redirect to the top page only (links to inner pages remain untouched)
  • Output hreflang on every page, explicitly stating the correspondence between language versions
  • Declare a default page via x-default for visitors who don’t match any language

All three of these tell Google which language versions exist and how they correspond — none of them eliminate the redirect itself on the top page. The concern Google raises — that search engines might be prevented from viewing all versions of the site — remains as long as the automatic redirect exists, regardless of how well hreflang is set up.

We should note that Search Console’s report alone gives no way to verify exactly what internal process led Google to replace /en/’s canonical. What we’ve laid out here is an inference based on cross-referencing Google’s published specifications and recommendations against our own implementation, not a confirmed causal chain.

Fix

We removed the top-page-only automatic redirect, along with the accompanying logic that stored a visitor’s language-switch click in localStorage and gave it priority over the automatic detection on subsequent visits.

Here is the logic that had been embedded in the top page’s <head> before removal.

;(() => {
  try {
    if (document.referrer && new URL(document.referrer).origin === location.origin) return
    const target = (l) => (l === 'en' ? '/en/' : l === 'zh' ? '/zh/' : null)
    const stored = localStorage.getItem('preferred-lang')
    if (stored) {
      const t = target(stored)
      if (t) location.replace(t)
      return
    }
    for (const l of navigator.languages || [navigator.language]) {
      const s = (l || '').toLowerCase()
      if (s.startsWith('ja')) return
      const t = target(s.slice(0, 2))
      if (t) { location.replace(t); return }
    }
  } catch {}
})()

After the removal, this logic no longer exists on the top page’s <head>. We also removed the script that stored language-switch clicks in localStorage, along with the data-lang-switch attribute that existed only to support it. Visitors are now guided to each language version solely through the ordinary language-switch links in the header and footer (plain links to /, /en/, and /zh/) and the hreflang tags on every page.

Summary

  • Google Search Console’s “Duplicate, Google chose different canonical than user” can occur even without any mistake in the canonical or hreflang tag implementation
  • Because Googlebot does not send an Accept-Language header when crawling, a redirect based on the browser’s language setting can behave unpredictably for Googlebot
  • Google explicitly recommends avoiding automatic redirects based on a guessed user language, and having hreflang in place does not substitute for following that recommendation
  • As a fix, we removed the top-page-only automatic redirect and its associated localStorage logic, keeping only the manual language-switch links and hreflang tags

References