Even the best-written service page doesn’t exist in search results if Google can’t crawl or index it. Technical SEO is the work of clearing the path between your content and Google, and most problems can be found with a few minutes of checking.
We prepared this article as an audit checklist. We grouped the items in the order of Google’s search process: first crawling, then indexing, then page experience and rich results. At the end of each group we also note which tool to check it with. Our primary source is the Google Search Central documentation.
If you’re curious about the non-technical pillars of SEO, our article What is SEO gives the overall framework. Here we go straight into the site’s infrastructure.
Before starting a technical SEO audit: free tools
You don’t need expensive software for a technical SEO audit. To start, these free tools do most of the job:
| Tool | What for? |
|---|---|
| Google Search Console | Indexing status, URL inspection, sitemaps, Core Web Vitals and manual action notices |
| PageSpeed Insights | Speed measurement with real user data (CrUX) and lab tests |
| Rich Results Test | Whether structured data is error-free and eligible for rich results |
| Browser developer tools | The page’s source code, redirects, HTTP status codes, meta tags |
| A site crawler | Crawling the whole site to find broken links, duplicate titles and redirect chains (the free version of many is enough for small sites) |
Crawling: can Google reach your site?
Googlebot being able to access your pages is the precondition for everything. Check:
- 1. Is robots.txt correct? Open yourdomain.com/robots.txt. A “Disallow: /” line left over from the development environment at launch closes the whole site to crawling. It is a surprisingly common mistake.
- 2. Are you not using robots.txt as a hiding tool? According to Google, robots.txt manages crawling; it doesn’t hide a page from search results. A blocked page that gets links from other sites can still appear in the index without its content. To hide a page, use noindex.
- 3. Are CSS and JavaScript files accessible? Google renders the page like a browser; if you block style and script files, it can’t see the page correctly.
- 4. Is your server or CDN not blocking Googlebot? Firewall or bot protection settings can mistake Googlebot for an attacker. Google’s AI features document also recommends that crawling be allowed not just in robots.txt but also in your CDN and hosting infrastructure.
- 5. Are important pages reachable via internal links? “Orphan” pages that receive no links from any page are hard to discover. Links should be plain a tags with an href; Google may not be able to follow links that work only through a JavaScript click event.
- 6. Are there no server errors (5xx)? A server that keeps returning errors causes Google to slow its crawl rate. Check the crawl stats report in Search Console.
Indexing: are the right pages in the index?
Not every crawled page needs to be indexed; thank-you pages, filter combinations, the basket and the admin panel shouldn’t be. What matters is that the right pages are indexed.
- 7. Is there no accidental noindex on important pages? The pre-launch “discourage search engines” option may be left on by mistake. Check your main pages one by one with the URL Inspection tool.
- 8. Is each piece of content at a single URL? If http and https, www and non-www, and trailing-slash and non-slash URLs show the same page, they should all be 301-redirected to a single version.
- 9. Are canonical tags correct? Each page should point to itself (or the preferred main version) as canonical. Google wants the canonical tag, internal links and sitemap to point to the same URL; conflicting signals lead Google to make its own choice.
- 10. Is the sitemap clean? The sitemap should contain only canonical URLs that return 200 and that you want indexed. A single file can contain at most 50,000 URLs or 50 MB uncompressed; large sites use a sitemap index.
- 11. Are lastmod dates real? Google uses the lastmod value only if it is accurate and wants it to reflect a significant update. The priority and changefreq fields are ignored by Google.
- 12. Has the sitemap been submitted to Search Console and referenced in robots.txt?
- 13. Are thin and duplicate pages under control? Dozens of pages with nearly identical text waste crawl budget and weaken the quality signal. Merge them, improve them or noindex them.
URL structure, HTTPS and error pages
- 14. Are URLs readable? Meaningful, short, lowercase URLs like “/services/corporate-website” are easier for both users and Google to understand than parameterised URLs like “/page.php?id=482&cat=7”. Using plain Latin letters instead of Turkish characters prevents them from breaking when shared.
- 15. Is the whole site on HTTPS? You need a valid SSL certificate and no mixed content warnings (http images or scripts on an https page). HTTPS is part of Google’s page experience assessment.
- 16. Do deleted pages return the right code? A permanently removed page should return 404 or 410; if there is a similar replacement page, it should be 301-redirected there. Non-existent pages showing empty content with “200 OK” (soft 404) mislead Google.
- 17. Are there no redirect chains? Shorten chains going from A to B and B to C so they go directly from A to C.
- 18. Do you have a user-friendly 404 page? A 404 page with a search box and links to the main categories keeps lost visitors on the site.
Mobile-first indexing and Core Web Vitals
Google completed the move to mobile-first indexing in 2024. This means that when evaluating your site, it primarily crawls and indexes the mobile version. Text that appears on desktop but is hidden on mobile is largely non-existent for Google too.
- 19. Do mobile and desktop offer the same content? Headings, text, images, structured data and meta tags should be the same in both versions.
- 20. Is there a viewport tag, is text readable without zooming, and are buttons easy to tap with a finger?
- 21. Are Core Web Vitals thresholds being met? According to web.dev, for a good experience LCP should be 2.5 seconds or less, INP 200 milliseconds or less and CLS 0.1 or less for 75 percent of visits. INP replaced FID in 2024.
- 22. Are images optimised? WebP or AVIF format, files at the size they are displayed, lazy loading for off-screen images and width and height values on every image.
Speed is a field of expertise in its own right; we explain how to improve LCP, INP and CLS in detail in our site speed and Core Web Vitals guide.
Technical SEO for JavaScript-driven sites
Sites built with frameworks like React, Vue or Angular that render content in the browser can be processed by Google; but this means an extra step and extra risk. Google’s JavaScript SEO documentation explains that pages are first crawled, then queued for rendering.
- 23. Is the main content in the initial HTML? Server-side rendering (SSR) or pre-rendering (static HTML) makes headings, text and links visible without waiting for JavaScript. This matters not just for Google but for many AI crawlers too; some of them don’t execute JavaScript.
- 24. Does each view have its own URL? In single-page applications, the URL should change when the content changes; content managed with fragments after “#” isn’t indexed as separate pages.
- 25. Does the “crawled page” view in URL Inspection show the content in full?
On the sites we build with custom software, we make these decisions from the start; you can see our approach on our web solutions page.
Structured data and multilingual sites
- 26. Are the appropriate schema types added? Markup such as Organization, LocalBusiness, Product, Article and BreadcrumbList helps Google understand the page and makes it eligible for rich results. The markup must match the content visible on the page exactly.
- 27. Are there no errors in the Rich Results Test? Also set expectations correctly: since 2023 Google shows FAQ rich results only for well-known, authoritative government and health sites. The markup still does no harm as a meaning signal, but it doesn’t bring a visual gain on every site.
- 28. Is hreflang correct on a multilingual site? If you have language versions such as Turkish, English and German, each page should reference its counterparts in other languages and itself with hreflang; the links must be reciprocal. Google recommends letting users choose a language rather than forcing language versions through automatic IP redirects.
Technical SEO won’t raise rankings on its own; but a technical error can make even the best content invisible.
Technical settings for AI features and crawlers
In 2026 a new heading was added to the technical SEO list: AI. Google’s “AI features and your website” document says there are no special technical requirements for AI Overviews and AI Mode; it is enough for the page to be indexed and eligible to be shown with a snippet. No new AI file or special markup is needed. Still, there are checks you should know about:
- 29. Are snippet controls used deliberately? The nosnippet, data-nosnippet and max-snippet directives limit how much of your content is shown in search results and AI summaries. An accidentally added nosnippet eliminates your page’s chance of being a source in summaries.
- 30. Is your policy for AI crawlers clear? The Google-Extended token lets you limit the use of your content in training models such as Gemini and in some other Google systems; it doesn’t affect visibility in Google Search. Other companies’ AI crawlers can also be managed individually by name in robots.txt. Blocking or allowing is a business decision; what matters is doing it knowingly.
In Search Console, traffic from AI features is included in the performance report for the “Web” search type. After fixing technical issues, you can track measurement there.
- No “Disallow: /” line in robots.txt closing off the site
- No noindex on important pages; the canonical points to itself
- http and non-www versions 301-redirect to a single URL
- Sitemap submitted and contains only URLs returning 200
- The mobile version offers the same content as desktop
- Core Web Vitals at “good” level on mobile
- Old URLs redirected one by one to new URLs
- Structured data passes the Rich Results Test without errors
Frequently asked questions
What is technical SEO?
Technical SEO is the infrastructure work that allows search engines to crawl, understand and index a website without problems. robots.txt, sitemaps, canonical tags, redirects, HTTPS, mobile-friendliness, page speed and structured data all fall within its scope.
How often should a technical SEO audit be done?
For small sites that change little, a general check every few months and regular monitoring of Search Console alerts are enough. Before and right after a site migration, redesign, domain change or major software update, however, a detailed audit is a must.
Why isn’t my page showing up on Google?
The most common reasons are that the page carries a noindex tag, is blocked by robots.txt, receives no links from any page, has another URL chosen as canonical due to duplicate content, or hasn’t been crawled yet. The URL Inspection tool in Search Console largely reveals the reason.
What is the difference between robots.txt and noindex?
robots.txt prevents Google from crawling a page, but if the page gets links from elsewhere it can appear in the index without its content. noindex, on the other hand, allows the page to be crawled and says it should not be shown in search results. To remove a page from results, use noindex and don’t block the page with robots.txt.
Is a special file like llms.txt needed for AI search?
According to Google’s official documentation, no new AI file or special markup is needed to appear in AI Overviews and AI Mode. It is enough for the page to be indexed and eligible to be shown with a snippet. How other AI services use such files varies.
Final word
The nice thing about technical SEO is that most problems are measurable and fixable. Running this checklist a few times a year and before every major change lets you catch early the errors that quietly erode your visibility. Once the technical foundation is solid, you can devote your effort to content and local visibility; if your business serves a specific region, the next step should be local SEO and Google Business Profile.
At CYF Tech, we check these items one by one before launch on the sites we build. If you’d like a technical audit of your current site, or to plan a site migration without losing visibility, contact us.
- Google Search Central — Introduction to robots.txt
- Google Search Central — Consolidating duplicate URLs (canonical)
- Google Search Central — Build and submit a sitemap
- Google Search Central — Mobile-first indexing
- Google Search Central — JavaScript SEO basics
- Google Search Central — Localized versions (hreflang)
- Google Search Central — Introduction to structured data
- Google Search Central — AI features and your website
- Google Search Central Blog — Changes to HowTo and FAQ rich results (2023)
- web.dev — Web Vitals