FAQ: How do you find new sites?I made my very own web crawler!
Also known as a "bot", it's an automated program that looks for interesting things and makes little reports about the websites it finds. It's not perfect, so I need to manually check that the reports were made properly. Once they've been checked and I've decided it's okay to list the websites they represent, other programs I've made bundle the completed reports up and upload them to the Website Finder's database.
Despite having to tweak the reports myself, it's actually a pretty hands off approach.
Back to the main page