In this document, you can find quotes for Crawling robots that I tried to compile in the very first time in doing some literature review and research. Therefore, it might not be the optimal thing, but I have learnt a lot from such process and my methods were improved a lot.

https://docs.google.com/document/d/1r8IgzEU3YmUibFW-5y7p_8tUSHI_SXJBhvdl99i3q9w/edit?usp=sharing