Designing distributed web crawler

requirements

  1. scalable - distributed and throttled to deal with rate limit
  2. performance - process tasks in parallel and be efficient in time and space
  3. compatibility - both static and dynamic sites
2 Likes