Notes and authoritative sources: Google’s crawl budget guidance (developers.google.com/crawling/docs/crawl-budget) explains when to worry about crawl budget; web.dev documents the Baseline availability of LCP and INP APIs (Dec 2025), which affects how CrUX collects field metrics for these Core Web Vitals.

Workflow

Step 1: Audit robots.txt

Check your public/robots.txt or generate it dynamically with Next.js. Make sure you do not accidentally block indexable pages (including API routes you rely on for prerendering). Example dynamic robots file for a Next.js app:

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL ?? "https://yourbrand.com";
  return {
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: ["/api/", "/admin/", "/_next/"],
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

Checklist items for robots.txt:

• Is the file at /robots.txt on the production domain?
• Does it accidentally disallow important paths (APIs used for SSR, sitemaps, or public content)?
• Does your staging/preview environment serve X-Robots-Tag:noindex and block preview URLs?

Step 2: Generate and validate sitemaps

Sitemaps tell Google what to crawl and when. Ensure sitemap(s):

• Are reachable at the production domain (https://yourbrand.com/sitemap.xml)
• Use a sitemap index if >50,000 URLs or >50MB (uncompressed)
• Use lastmod correctly for frequently updated content
• Match canonical URLs and do not include noindex pages

Example dynamic sitemap generator (Next.js):

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL ?? "https://yourbrand.com";
  const posts = await getAllPosts();

  const staticPages: MetadataRoute.Sitemap = [
    { url: baseUrl, lastModified: new Date(), changeFrequency: "daily", priority: 1.0 },
    { url: `${baseUrl}/about`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.5 },
  ];

  const postPages: MetadataRoute.Sitemap = posts.map((post) => ({
    url: `${baseUrl}/blog/${post.slug}`,
    lastModified: new Date(post.updatedAt),
    changeFrequency: "weekly" as const,
    priority: 0.7,
  }));

  return [...staticPages, ...postPages];
}

Step 3: Verify canonicalization

Ensure every page sets a self-referencing canonical via your framework's metadata APIs. Rules:

• Use the production domain and canonicalize to the preferred host (HTTPS, www or non-www) consistently
• Avoid referencing preview or localhost URLs in canonicals
• Exclude irrelevant query parameters unless they change page content
• Canonical in HTML must match the URL in sitemap.xml

Example (Next.js metadata):

export async function generateMetadata({ params }: Props): Promise<Metadata> {
  const { slug } = await params;
  return {
    alternates: {
      canonical: `https://yourbrand.com/blog/${slug}`,
    },
  };
}

Step 4: Find and fix redirect chains

Redirect chains slow discovery and waste crawl budget. Steps:

1. Crawl the site with Screaming Frog, Sitebulb, or a custom script
2. Identify chains >1 hop and replace the first redirect with a direct redirect to the final URL
3. Update internal links to point at the final destination

Next.js example config for clean redirects:

// next.config.ts
const nextConfig = {
  async redirects() {
    return [
      { source: "/old-path", destination: "/new-path", permanent: true },
      { source: "/legacy/:slug", destination: "/blog/:slug", permanent: true },
    ];
  },
};
export default nextConfig;

Step 5: Check rendering, SPA navigation, and indexability

Verify that core content is visible to Googlebot without requiring client-side navigation. Tests:

• Server-side rendering check: curl -s https://yourbrand.com/blog/my-post | head -100

• robots.txt and sitemap accessibility: curl -s https://yourbrand.com/robots.txt curl -s https://yourbrand.com/sitemap.xml | head -50

• HTTP header checks (X-Robots-Tag): curl -I https://yourbrand.com/blog/my-post 2>/dev/null | rg -i "x-robots"

Single-page apps and navigation:

• The Navigation API is now widely available (see web.dev), which affects how client-side navigation is handled. If you use SPA navigation, ensure each navigable URL is reachable as a unique URL (server-rendered snapshot or HTML snapshot for crawlers) and that pushState/History navigation updates metadata (title, canonical) correctly.
• For Next.js and hybrid apps, prefer server rendering or prerendering for important pages, and provide HTML snapshots for crawler-friendly navigation.

Indexing API and live content:

• Google has expanded its tooling around indexing and live content in recent Search Central communications (see Search Central Live recaps). Use the Indexing API where supported (jobs, certain live content) and follow Search Central guidance for structured data on livestreams and similar real-time content.

Step 6: Core Web Vitals audit (field + lab)

Field metrics (CrUX) now have broader browser coverage for LCP and INP following Baseline availability in major browsers (web.dev, Dec 2025). Audit both lab and field data:

• LCP (Largest Contentful Paint): target < 2.5s (field median for good) — fixes: optimize images, text-rendering, critical CSS, server TTFB, preconnect/preload critical assets

• CLS (Cumulative Layout Shift): target < 0.1 — fixes: set dimensions for images/media, reserve space for ads, avoid inserting content above the fold

• INP (Interaction to Next Paint): target < 200ms — fixes: reduce main-thread work, split long tasks, defer non-critical scripts, use web workers

Measurement tips:

• Use CrUX data (Search Console Core Web Vitals report) for field behavior and Lighthouse/DevTools for lab reproduction
• Because LCP and INP are now available as browser APIs, ensure your performance monitoring (RUM) captures the appropriate events (Largest Contentful Paint API and Event Timing API / INP polyfills where necessary)

Examples

Example 1: robots.txt for a SaaS site

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /_next/
Disallow: /dashboard/

Sitemap: https://yourbrand.com/sitemap.xml

Example 2: Middleware for trailing slash normalization

// middleware.ts
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";

export function middleware(request: NextRequest) {
  const { pathname } = request.nextUrl;
  if (pathname !== "/" && pathname.endsWith("/")) {
    const url = request.nextUrl.clone();
    url.pathname = pathname.slice(0, -1);
    return NextResponse.redirect(url, 308);
  }
}

export const config = { matcher: "/((?!api|_next|favicon.ico).*)" };

Decision tree

• New site launch → set up robots.ts, sitemap.ts, canonical URLs, and verify SSR output
• Traffic drop → check Search Console index coverage report and the Core Web Vitals report for field data
• Domain migration → create a comprehensive redirect map and verify in Search Console
• Slow pages → audit Core Web Vitals; start with LCP, then CLS, then INP
• Pages not indexing → check robots.txt, noindex headers, canonical conflicts, and crawl budget
• Duplicate content → implement canonical URLs and consolidate thin pages

Edge cases and gotchas

1. Preview/staging URLs indexed — search engines may index Vercel preview URLs. Add X-Robots-Tag: noindex header for non-production deployments via middleware.
2. Soft 404s — pages that return 200 but show error content. Google treats these as low quality. Return proper 404 status codes via framework handlers (notFound() in Next.js).
3. Sitemap size limits — sitemaps have a 50,000 URL / 50MB limit. Use a sitemap index for large sites.
4. hreflang conflicts — hreflang annotations must be bidirectional.
5. robots.txt cached by Google — changes may take hours to days to propagate; use Search Console to request re-crawl if urgent (see Google Search Central crawl budget and robots guidance).
6. Redirect loops — a → b → a creates an infinite loop. Always test redirect maps before deploying.

Evaluation criteria

• robots.txt exists and correctly allows/disallows paths
• sitemap.xml is generated dynamically and includes all indexable URLs
• Every page has a self-referencing canonical URL
• No redirect chains longer than one hop
• SSR output is visible without JavaScript execution for critical pages
• Core Web Vitals pass (LCP < 2.5s, CLS < 0.1, INP < 200ms) measured in field and lab
• No preview/staging URLs are indexable in production
• Search Console shows no critical indexing errors