We recently removed over 1 million duplicate articles from our Content Explorer index. The issue occurred for various reasons including:
- web pages containing mobile versions
- AMP pages
- our crawler visited the http and https version of the website (webmasters using non-canonical urls 🙄)
- the same article appeared multiple times throughout our crawls under different URL structures and permalinks
We’ve patched our system to prevent this from happening in future crawls so you shouldn’t have to worry about duplicates anymore. We appreciate all the feedback and support that y’all have given us so far. As always, feel free to email us if you have any issues or suggestions. ☺️
Please note, you may continue searching and downloading articles during this time, however, new articles are still being re-populated. Some domain searches may be unavailable during this time.