Ranking rich mobile verticals based on clicks and abandonment

We consider the problem of ranking rich verticals, which we call "cards," for a given mobile search query. Examples of card types in-clude "SHOP" (showing access and contact information of a shop), "WEATHER" (showing a weather forecast for a particular loca-tion), and "TV" (showing information about a TV programme). These cards can be highly visual and/or concise, and may often sat-isfy the user's information need without making her click on them. While this "good abandonment" of the search engine result page is ideal especially for mobile environments where the interaction be-tween the user and the search engine should be minimal, it poses a challenge for search engine companies whose ranking algorithms rely heavily on click data. In order to provide the right card types to the user for a given query, we propose a graph-based approach which extends a click-based automatic relevance estimation algo-rithm of Agrawal et al., by incorporating an abandonment-based preference rule. Using a real mobile query log from a commer-cial search engine, we constructed a data set containing 2,472 pair-wise card type preferences covering 992 distinct queries, by hiring three independent assessors. Our proposed method outperforms a click-only baseline by 53-68% in terms of card type preference accuracy. The improvement is also statistically highly significant, with p0:0000 according to the paired randomisation test.