123ArticleOnline Logo
Welcome to 123ArticleOnline.com!
ALL >> Education >> View Article

How To Check Which Urls Have Been Indexed Without Upsetting Google

Profile Picture
By Author: Aleisha
Total Articles: 1
Comment this article
Facebook ShareTwitter ShareGoogle+ ShareTwitter Share

How can I learn which pages have not been indexed by Google, and do so in a way that has not infringed Google's rules? Google does not indicate whether a page has been indexed in the Google Search Console, does not let us erase the search results to get the answer and does not want to indirectly get the response from an undocumented API.

Use Google Query Explorer and download it as tsv
You can then download your XML sitemap locally and open it in Excel. Then, drag it into the Excel window, and you will get the "Import XML" dialog box. If it asks you to "Open the file without applying a style sheet", select OK:

Importing an xml sitemap into excel
Next, choose to open the "As an XML table" file:

Import xml as table in excel
You can delete the foreign columns by retaining only the column "ns1: loc" (or "loc"):

Remove xml sitemap irrelevant columns after importing to Excel
Then you just need to do a VLOOKUP or some other form of Excel matching and find the URLs in the sitemap that are not present in the analysis data.

I thought it was a simple but smart solution, and although ...
... a good starting point, I was afraid it did not show exactly which pages were indexed by Google. It is not uncommon for pages to receive little or no traffic, even if they are indexed. It may be an indication that the page is not indexed, but it can also show that the page has a marking problem, has become useless, needs some optimization to improve its visibility or simply is not Present in the XML Sitemap.

The log file solution:
Server log files are an excellent source of data on your website that is often inaccessible by other means. One of the many information that can be derived from these log files is whether or not a certain bot has accessed your website. In our case, the bot to which we are concerned is Googlebot.

Scanning our server log files allows us to check if Googlebot has already visited a certain page on our website. If Googlebot has never visited a certain page, it may not have been indexed by Google. I personally tend to use KNIME for this purpose, with the integrated Web Log Reader node, but do not hesitate to use your preferred solution.

Screaming Frog Log File Analyzer provides an easier solution for log file analysis.
Screaming Frog Log File Analyzer provides an easier solution for log file analysis.
Like Google Analytics, scanning log files is not foolproof. Googlebot may visit a page but do not include it in its index.

Combining your data
To restrict our list of pages that can not be indexed by Google as much as possible, I recommend combining the captured data using the Google Analytics technique with the log file analysis methods above.

Conclusion
Given that Google does not provide a tool or data on whether a web page has been indexed or not, and we are not allowed to use an automated solution like the one I wrote previously, we must Try to narrow down our list of URLs that may not be indexed.

We can do this by reviewing our Google Analytics data for pages on our website that do not receive organic Google traffic and view server log files. From there, we can manually check our short URL list punctually.

It's not an ideal solution, but it does the job. I hope that in the future, Google will provide a better way to evaluate which pages have been indexed and which ones have not.

Total Views: 297Word Count: 590See All articles From Author

Add Comment

Education Articles

1. Devops: The Modern Skillset Every Tech Professional Should Master
Author: safarisprz01

2. Salesforce Marketing Cloud Training In India | Cloud
Author: Visualpath

3. How An English Medium School Shapes A Child’s Future In Today’s Global World
Author: Mount Litera Zee School

4. Mern Stack Online Training In Ameerpet | Mern Stack Ai Training
Author: Hari

5. Why Online Courses In Sap Sd Are The Best Solution For Today's Professionals
Author: ezylern

6. Sailpoint Online Course In Bangalore For Professionals
Author: Pravin

7. Sap Ai Course | Sap Ai Online Training In Hyderabad
Author: gollakalyan

8. Why Aima Is The Best Choice For A Global Advanced Management Programme
Author: Aima Courses

9. The Best Oracle Integration Cloud Online Training
Author: naveen

10. Mlops Training Course In Chennai | Mlops Training
Author: visualpath

11. International Cbse School In Nallagandla,
Author: Johnwick

12. Best Mba Dual Specialization Combinations For 2025 And Beyond
Author: IIBMS Institute

13. Top Docker Kubernetes Training In Hyderabad | Docker And Kubernetestop Docker Kubernetes Training In Hyderabad | Docker And Kubernetes
Author: krishna

14. Full Stack Web Development Course In Noida
Author: Training Basket

15. Master Advanced Pega Skills With Pega Cssa Infinity'24.2 Online Training By Pegagang
Author: PegaGang

Login To Account
Login Email:
Password:
Forgot Password?
New User?
Sign Up Newsletter
Email Address: