Updated: Feb 17, 2020

Crowdsensus logo, the letter C made out of a number of avatars converging


End-user elicitation studies are a popular design method. Currently, such studies are usually confined to a lab, limiting the number and diversity of participants, and therefore the representativeness of their results. Furthermore, the quality of the results from such studies generally lacks any formal means of evaluation. In this paper, we address some of the limitations of elicitation studies through the creation of the Crowdlicit system along with the introduction of end-user identification studies, which are the reverse of elicitation studies. Crowdlicit is a new web-based system that enables researchers to conduct online and in-lab elicitation and identification studies. We used Crowdlicit to run a crowd-powered elicitation study based on Morris’s “Web on the Wall” study (2012) with 78 participants, arriving at a set of symbols that included six new symbols different from Morris’s. We evaluated the effectiveness of 49 symbols (43 from Morris and six from Crowdlicit) by conducting a crowd-powered identification study. We show that the Crowdlicit elicitation study resulted in a set of symbols that was significantly more identifiable than Morris’s.

Research Areas

Design; Crowdsourcing.

My Involvement

I built a platform to run end-user elicitation studies and deploy it online to collect interaction designs from users all over the world.

The tool was created using HTML, Javascript, PHP, and MySQL.

I designed a series of experiments to test the validity and effectiveness of tool.

I helped formulate the end-user identification study method—the reverse of end-user elicitation study.

I handled all the data collection and analysis.

I was the lead author on the CHI 2019 paper.


University of Washington


Abdullah X. Ali, Meredith Ringel Morris, Jacob O. Wobbrock. 2019. Crowdlicit: A System for Conducting Distributed End-User Elicitation and Identification Studies. In 2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland, UK. ACM, New York, NY, USA.


Presentation Talk

#Projects #All #Crowdsourcing