Computer Science

Googlebot-Image

Google crawls the site for images and indexes the images.

Googlebot-Video

It crawls the site to make videos appear in Google Video searches.

Roles in robots.txt


• These lines allow these bots to access the entire site.
• This means you are allowing Google to crawl and index images and videos.

                        User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Video
Allow: /