TY  - JOUR
A1  - Plachouras, Vassilis
A1  - Carpentier, Florent
A1  - Faheem, Muhammad
A1  - Masanès, Julien
A1  - Risse, Thomas
A1  - Senellart, Pierre
A1  - Siehndel, Patrick
A1  - Stavrakas, Yannis
T1  - ARCOMEM crawling architecture
T2  - Future Internet
N2  - The World Wide Web is the largest information repository available today. However, this information is very volatile and Web archiving is essential to preserve it for the future. Existing approaches to Web archiving are based on simple definitions of the scope of Web pages to crawl and are limited to basic interactions with Web servers. The aim of the ARCOMEM project is to overcome these limitations and to provide flexible, adaptive and intelligent content acquisition, relying on social media to create topical Web archives. In this article, we focus on ARCOMEM’s crawling architecture. We introduce the overall architecture and we describe its modules, such as the online analysis module, which computes a priority for the Web pages to be crawled, and the Application-Aware Helper which takes into account the type of Web sites and applications to extract structure from crawled content. We also describe a large-scale distributed crawler that has been developed, as well as the modifications we have implemented to adapt Heritrix, an open source crawler, to the needs of the project. Our experimental results from real crawls show that ARCOMEM’s crawling architecture is effective in acquiring focused information about a topic and leveraging the information from social media.
KW  - web archiving
KW  - crawling architecture
KW  - content acquisition
Y1  - 2014
UR  - http://publikationen.ub.uni-frankfurt.de/frontdoor/index/index/docId/53651
UR  - https://nbn-resolving.org/urn:nbn:de:hebis:30:3-536511
SN  - 1999-5903
N1  - This is an open access article distributed under the Creative Commons Attribution License
VL  - 6
IS  - 3
SP  - 518
EP  - 541
PB  - MDPI
CY  - Basel
ER  -