# robots.txt 入门

> 来源：Google Search Central (https://developers.google.com/search/docs/crawling-indexing-robots-intro.txt)

---

* Ranking and search appearance
  ** Visual Elements gallery  **
  ** Title links  **
  ** Snippets  **
  ** Images  **
  ** Videos  **
  ** Structured data  **
  ** Favicons  **

  * Site-specific guides
  ** Ecommerce  **
  ** International and multilingual sites  **

  * Data analysis
  ** Get started with Search Console  **
  ** Debug traffic drops  **
  ** Using Search Console and Google Analytics data for SEO  **
  ** Get started with Google Trends  **

[ Support ](https://developers.google.com/search/help) [ Blog ](https://developers.google.com/search/blog) [ What's new ](https://developers.google.com/search/news)

  ** All updates  **
  ** Documentation updates  **
  ** Ranking updates  **
  ** New YouTube videos  **
  ** Recent podcast episodes  **

[ Events ](https://developers.google.com/search/events) [ Case studies ](https://developers.google.com/search/case-studies)

`/`

  * English
  * Deutsch
  * Español
  * Español – América Latina
  * Français
  * Indonesia
  * Italiano
  * Polski
  * Português – Brasil
  * Tiếng Việt
  * Türkçe
  * Русский
  * العربيّة
  * हिंदी
  * ภาษาไทย
  * 中文 – 简体
  * 中文 – 繁體
  * 日本語
  * 한국어

Sign in

  ** Documentation **

[Search Console](https://goo.gle/searchconsole)

[ ](https://developers.google.com/search)

  ** Google Search Central **

  ** Documentation  **
    * More 
  ** Support  **
  ** Blog  **
  ** What's new  **
    * More 
  ** Events  **
  ** Case studies  **

  **Introduction**
  * Search Essentials

    **Overview**
    **Technical requirements**
    **Spam policies**

  * SEO fundamentals

    **SEO Starter Guide**
    **How Google Search works**
    **Creating helpful, reliable, people-first content**
    **Do you need an SEO?**
    **Maintaining your site's SEO**
    **Developer's guide to Search**
    **Get your website on Google**
    **Guidance on using generative AI**

  * Crawling and indexing

    **Overview**
    **File types Google can index**
    **URL structure**
    **Links**
    * Sitemaps

      **Learn about sitemaps**
      **Build and submit a sitemap**
      **Manage sitemaps with sitemap index file**
      * Sitemap extensions

        **Image sitemaps**
        **News sitemaps**
        **Video sitemaps and alternatives**
        **Combining sitemap extensions**

    * Crawler management

      **Ask Google to recrawl your URLs**
      **Troubleshoot crawl errors**
      * Google crawlers

        **List of Google crawlers**
        **Googlebot**
        **Reduce the Google crawl rate**
        **Verifying Googlebot and other Google crawlers**

    * robots.txt

      **Introduction to robots.txt**
      **How Google interprets the robots.txt specification**

    * Canonicalization

      **What is URL canonicalization**
      **How to specify a canonical URL with rel="canonical" and other methods**
      **Fix canonicalization issues**

    **Mobile site and mobile-first indexing**
    * AMP

      **AMP on Google Search guidelines**
      **Understand how AMP works in search results**
      **Enhance AMP content**
      **Validate AMP content**
      **Remove AMP content**

    * JavaScript

      **Understand the JavaScript SEO basics**
      **Fix search-related JavaScript problems**
      **Fix lazy-loaded content**
      **Dynamic rendering as a workaround**

    * Page and content metadata

      **Page metadata**
      * Meta tags

        **Meta tags and HTML attributes that Google supports**
        **Robots meta tag, data-nosnippet, and X-Robots-Tag**
        **noindex**

      **rel attributes**

    * Removals

      **Control what you share with Google**
      **Page removals**
      **Image removals**
      **Redacted information**

    * Site moves and changes

      **Redirects and Google Search**
      * Site moves

        **Changing your hosting**
        **Move a site with URL changes**

      **A/B testing**
      **Temporarily pause or disable a website**

  * Ranking and search appearance

    **Overview**
    **AI features**
    **Byline dates**
    **Favicons**
    **Featured snippets**
    **Flexible Sampling**
    **Google Discover**
    **Images**
    * Local features

      **Business details**
      **Top Places List**
      **Opting out of local search results**

    * Page experience

      **Understanding page experience**
      **Core Web Vitals**
      **Interstitials and dialogs**
      **Get started with signed exchanges on Google Search**

    **Preferred sources**
    * Ranking systems

      **A guide to Google Search ranking systems**
      **Reviews system**

    * Ranking updates

      **List of ranking updates**
      **Core updates**
      **Spam updates**

    **Site names**
    **Sitelinks**
    **Snippets**
    * Structured data

      **Understand how structured data works**
      **Structured data general guidelines**
      **Enriched search results**
      **Generate structured data with JavaScript**
      * Feature guides

        **All structured data features**
        **Article**
        **Book actions**
        **Breadcrumb**
        **Carousel**
        **Course list**
        **Dataset**
        **Discussion forum**
        **Education Q&A**
        **Employer aggregate rating**
        **Fact check**
        **Event**
        **FAQ**
        **Image metadata**
        **Job posting**
        **Local business**
        **Math solver**
        **Movie carousel**
        **Organization**
        * Shopping

          **Overview**
          **Product snippet**
          **Merchant listing**
          **Variants**
          **Loyalty program**
          **Merchant return policy**
          **Merchant shipping policy**

        **Profile page**
        **Q&A**
        **Recipe**
        **Review snippet**
        **Software app**
        **Speakable**
        **Subscription and paywalled content**
        **Vacation rental**
        **Video**

    **Title links**
    * Translated features

      **Translated results**
      **Ad networks and translation-related Google Search features**

    **Videos**
    **Visual Elements gallery**
    * Web Stories

      **Enable Web Stories on Google**
      **Best practices for creating Web Stories**
      **Web Story content policy**

    * Early Adopters Program

      **Package tracking**
      **Structured data carousels (beta)**

  * Monitoring and debugging

    **Debug drops in Search traffic**
    * Monitor with Search Console

      **Get started with Search Console**
      **Improve SEO with a bubble chart**
      **Using Search Console and Google Analytics data for SEO**

    * Debug with search operators

      **Overview**
      **site: search operator**
      **Google Images search operators**

    * Preventing and monitoring abuse

      **Overview**
      **Prevent user-generated spam**
      **Malware and unwanted software**
      **Prevent a malware infection**
      **Social engineering (phishing and deceptive sites)**
      **Google Safe Browsing Repeat Offenders Policy**

    **Get started with Google Trends**

  * Site-specific guides

    * Ecommerce

      **Overview**
      **Where content can appear**
      **Share product data**
      **Include structured data**
      **Launch a new website**
      **Write high quality reviews**
      **Design a URL structure**
      **Ecommerce site structure**
      **Pagination, incremental page loading, and Search**

    * International and multilingual

      **Overview**
      **Managing multi-regional and multilingual sites**
      **Tell Google about localized versions of your page**
      **How Google crawls locale-adaptive pages**

    * Explicit content

      **Guidelines for sites with explicit content**
      **What to do if your site is incorrectly flagged as explicit**

  * SEO fundamentals 
  ** Introduction  **
  ** Search Essentials  **
  ** SEO Starter Guide  **
  ** How Google Search Works  **
  ** Do you need an SEO?  **
  * Crawling and indexing 
  ** Sitemaps  **
  ** robots.txt  **
  ** Meta tags  **
  ** Crawler management  **
  ** Removals  **
  ** Canonicalization  **
  ** Redirects  **
  ** JavaScript SEO  **
  * Ranking and search appearance 
  ** Visual Elements gallery  **
  ** Title links  **
  ** Snippets  **
  ** Images  **
  ** Videos  **
  ** Structured data  **
  ** Favicons  **
  * Site-specific guides 
  ** Ecommerce  **
  ** International and multilingual sites  **
  * Data analysis 
  ** Get started with Search Console  **
  ** Debug traffic drops  **
  ** Using Search Console and Google Analytics data for SEO  **
  ** Get started with Google Trends  **

  ** All updates  **
  ** Documentation updates  **
  ** Ranking updates  **
  ** New YouTube videos  **
  ** Recent podcast episodes  **

  ** Home **
  ** Search Central **
  ** Documentation **

Send feedback 

# Introduction to robots.txt

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, [block indexing with `noindex`](/search/docs/crawling-indexing/block-indexing) or password-protect the page. 

If you use a CMS, such as Wix or Blogger, you might not need to (or be able to) edit your robots.txt file directly. Instead, your CMS might expose a search settings page or some other mechanism to tell search engines whether or not to crawl your page. 

If you want to hide or unhide one of your pages from search engines, search for instructions about modifying your page visibility in search engines on your CMS (for example, search for "wix hide page from search engines"). 

## What is a robots.txt file used for?

A robots.txt file is used primarily to manage crawler traffic to your site, and usually to keep a file off Google, depending on the file type: 

robots.txt effect on different file types  
---  
Web page |  You can use a robots.txt file for web pages (HTML, PDF, or other [non-media formats that Google can read](/search/docs/crawling-indexing/indexable-file-types)), to manage crawling traffic if you think your server will be overwhelmed by requests from Google's crawler, or to avoid crawling unimportant or similar pages on your site.  Warning: Don't use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google Search results.  If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or [`noindex`](/search/docs/crawling-indexing/block-indexing).  If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result [won't have a description](https://support.google.com/webmasters/answer/7489871). Image files, video files, PDFs, and other non-HTML files embedded in the blocked page will be excluded from crawling, too, unless they're referenced by other pages that are allowed for crawling. If you see this search result for your page and want to fix it, remove the robots.txt entry blocking the page. If you want to hide the page completely from Search, use [another method](/search/docs/crawling-indexing/remove-information#i-control-the-web-page).   
Media file |  Use a robots.txt file to manage crawl traffic, and also to prevent image, video, and audio files from appearing in Google Search results. This won't prevent other pages or users from linking to your image, video, or audio file. 

  **Read more about preventing images from appearing on Google.**
  **Read more about how to remove or restrict your video files from appearing on Google.**

  
Resource file |  You can use a robots.txt file to block resource files such as unimportant image, script, or style files, if you think that pages loaded without these resources won't be significantly affected by the loss. However, if the absence of these resources make the page harder for Google's crawler to understand the page, don't block them, or else Google won't do a good job of analyzing pages that depend on those resources.   
  
## Understand the limitations of a robots.txt file

Before you create or edit a robots.txt file, you should know the limits of this URL blocking method. Depending on your goals and situation, you might want to consider other mechanisms to ensure your URLs are not findable on the web. 

  * robots.txt rules may not be supported by all search engines.  
The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not. Therefore, if you want to keep information secure from web crawlers, it's better to use other blocking methods, such as [password-protecting private files on your server](/search/docs/crawling-indexing/control-what-you-share). 
  * Different crawlers interpret syntax differently.  
Although respectable web crawlers follow the rules in a robots.txt file, each crawler might interpret the rules differently. You should know the [proper syntax](/search/docs/crawling-indexing/robots/robots_txt#syntax) for addressing different web crawlers as some might not understand certain instructions. 
  * A page that's disallowed in robots.txt can still be indexed if linked to from other sites.  
While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results. To properly prevent your URL from appearing in Google Search results, [password-protect the files on your server](/search/docs/crawling-indexing/control-what-you-share), [use the `noindex` `meta` tag or response header](/search/docs/crawling-indexing/block-indexing), or remove the page entirely. 

Caution: Combining multiple crawling and indexing rules might cause some rules to counteract other rules. Learn how to [combine crawling with indexing and serving rules](/search/docs/crawling-indexing/robots-meta-tag#combining). 

## Create or update a robots.txt file

If you decided that you need one, learn how to [create a robots.txt file](/search/docs/crawling-indexing/robots/create-robots-txt). Or if you already have one, learn how to [update it](/search/docs/crawling-indexing/robots/submit-updated-robots-txt). 

Want to learn more? Check out the following resources: 

  **How to write and submit a robots.txt file**
  **Update your robots.txt file**
  **How Google interprets the robots.txt specification**

Send feedback 

Except as otherwise noted, the content of this page is licensed under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/), and code samples are licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). For details, see the [Google Developers Site Policies](https://developers.google.com/site-policies). Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-10 UTC.

Need to tell us more?  [[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-12-10 UTC."],[],["A robots.txt file manages search engine crawler access to URLs on a site, primarily to avoid server overload. It's not for hiding pages from Google; use `noindex` or password protection for that. The robots.txt file can also manage crawl traffic for media files to prevent them from appearing in search results. However, pages disallowed in robots.txt can still be indexed if linked externally, and it may not be supported by all search engines. Consider alternative methods like `noindex` or password protection.\n"]] 

  ** LinkedIn  **

Join us on LinkedIn

  ** YouTube  **

Watch our videos

  ** Blog  **

Subscribe to our RSS feed

  ** Podcast  **

Listen to Search Off the Record

  ** X (Twitter)  **

Join us on X (Twitter)

  * ### Get support

    ** Go to the help forum **
    ** Submit a question for office hours **
    ** Report spam, phishing, or malware **
    ** More support resources **
  * ### Resources

    ** Do you need an SEO? **
    ** SEO Starter Guide **
    ** Status of Search systems **
    ** Search Console documentation **
    ** Case Studies **
  * ### Tools

    ** Search Console **
    ** Rich Results Test **
    ** PageSpeed Insights **
    ** AMP Test **

[ ](https://developers.google.com/)

  ** Android **
  ** Chrome **
  ** Firebase **
  ** Google Cloud Platform **
  ** Google AI **
  ** All products **

  ** Terms **
  ** Privacy **
  * Manage cookies 

  * English
  * Deutsch
  * Español
  * Español – América Latina
  * Français
  * Indonesia
  * Italiano
  * Polski
  * Português – Brasil
  * Tiếng Việt
  * Türkçe
  * Русский
  * العربيّة
  * हिंदी
  * ภาษาไทย
  * 中文 – 简体
  * 中文 – 繁體
  * 日本語
  * 한국어
