Adaptive Focused Website Segment Crawler

Tanaphol Suebchua, Arnon Rungsawang, Hayato Yamana

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)


Focused web crawler has become indispensable for vertical search engines that provide a search service for specialized datasets. These vertical search engines have to collect specific web pages in the web space, whereas search engines such as Google and Bing gather web pages from all over the world. The problem in focused crawling research is how to collect specific web pages with minimal computing resources. We previously addressed this problem by proposing a focused crawling strategy, which utilizes an ensemble machine learning classifier to find the group of relevant web pages, referred to as relevant website segment. In this paper, we enhance the proposed crawler as follows: 1) We increase the accuracy of predicting website segments, by preparing two predictors: a predictor learned by features extracted from relevant source website segments and another predictor learned by features from irrelevant ones. The idea is that there may exist different characteristics between these two types of source website segments. 2) We also propose a noisy data elimination method when updating the predictor incrementally during the crawling process. A preliminary experiment shows that our enhanced crawler outperforms a crawler that equips neither of these approaches by around 12%, at most.

Original languageEnglish
Title of host publicationNBiS 2016 - 19th International Conference on Network-Based Information Systems
EditorsFatos Xhafa, Tomoya Enokido, Leonard Barolli, Makoto Takizawa, Vaclav Snasel
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)9781509009794
Publication statusPublished - 2016 Dec 16
Event19th International Conference on Network-Based Information Systems, NBiS 2016 - Ostrava, Czech Republic
Duration: 2016 Sept 72016 Sept 9

Publication series

NameNBiS 2016 - 19th International Conference on Network-Based Information Systems


Other19th International Conference on Network-Based Information Systems, NBiS 2016
Country/TerritoryCzech Republic


  • classifier ensemble
  • focused crawler
  • machine learning
  • noise reduction
  • topic specific web crawler
  • website segment crawler

ASJC Scopus subject areas

  • Information Systems
  • Computer Networks and Communications


Dive into the research topics of 'Adaptive Focused Website Segment Crawler'. Together they form a unique fingerprint.

Cite this