APIs & Crawl Budget: Don’t block API requests if they load important content
An attendee asked whether a website should disallow subdomains that are sending API requests, as they seemed to be taking up a lot of crawl budget. They also asked how API endpoints are discovered or used by Google.
You could help avoid crawl budget issues here by making sure the API results are cached well and don’t contain timestamps in the URL. If you don’t care about the content being returned to Google, you could block the API subdomains from being crawled, but you should test this out first to make sure it doesn’t stop critical content from being rendered.
John suggested making a test page that doesn’t crawl the API, or uses a broken URL for it, and see how the page renders in the browser (and for Google).
Image sitemaps can be useful for sites that use lazy loading
When “lazy loading” images on a page in a way that doesn’t include defined image elements, it’s recommended to have back-up in the form of structured data or an image sitemap. That way, Google will know to associate those images with the page even before they’re loaded.
Fetch & Render Tool in GSC Doesn’t Reflect Real Rendering
Getting ‘temporarily unreachable’ messages in the Fetch & Render tool doesn’t reflect how Google is rendering content for its index. Google’s rendering service has a longer cutoff time and uses caching.
Scroll Events Shouldn’t be Used in Isolation to Execute Lazy-loading
Scroll events aren’t always the best solution because they are expensive, users on desktop may resize their window to get more content which wouldn’t trigger a scroll event, and Google doesn’t scroll. Test lazy-loading is working by using Fetch & Render and Intersection Observer.
Text That’s Hidden by Default During Rendering is Fine For Google
Some sites will prevent content from being visible until the page has finished rendering to stop elements from jumping around the screen as they are loaded. This is fine for Google as long as the textual content is in the HTML, but check what Google can see with the mobile-friendly testing tool and Fetch & Render in GSC.
Google’s Cache Isn’t an Accurate View of Googlebot Rendering
The Fetch & Render tool in GSC and the Mobile-friendly Test tool show a more accurate view of how Googlebot is able to render a page than Google’s cache view, as this can easily be broken.
GSC Will Still Have the Option to Fetch Desktop Pages After Mobile-first Indexing
Google still wants website owners to be able to check how their desktop pages appear even if the content is being taken from mobile for mobile-first indexing, so there will still be an option to fetch both page versions in GSC.
Ensure Google is Shown the Same Title When the Page is Fetched & Rendered
If Google is switching the titles between individual URLs, then something with the back-end of the website may be wrong. Google should be able to get the same title when it initially fetches the page as when it is rendered.