Creating a statistically valid research protocol in order to measurably increase the quality of content creation on a comic journalism website.
Project Goal & Overview
My colleagues and I were interested in applying the benefits of the comic strip medium (quick stories, information in text & image) to news consumption by creating an online platform where individuals can create comic-strips from news articles.
I was approached to develop a procedure which would encourage users to create higher quality comic strips. After researching, I realized that there are four commonly used design methods which increase quality:
- Generating multiple design directions and then pursuing very few for further work.
- Breaking the content creation into smaller stages which are completed separately.
- Having an expert select the best designs for further iteration.
- Iterating on a particular design by creating close variants.
Method: Iterative Design on Amazon Mechanical Turk
I created a series of studies to investigate comic quality when these variables were incorporated versus excluded from the comic creation process. Using Amazon Mechanical Turk, I created two conditions – in one, the participants were asked to create a comic from a news article; the comic was later evaluated. In the second, the participants were asked to create a Narrative Storyboard of a comic based off of the news article; I solicited evaluations (from colleagues and other participants) to select a subset, which were then turned into comic. Those comics were later evaluated. A summary of this method appears at right.
Results: Statistically Higher Comic Quality
The results showed a that comic quality is directionally, but non-significantly increased by using a structure that implements the methods above. Interestingly, I noticed that within the results, the comics which were created without any structure (control condition) were more often off-topic, creative, and funny. In contrast, comics which were created with the structured approach (experimental condition) seemed more informational and higher quality, but were drier in tone. We tested this hypothesis in a follow-up study and found that comics created with the non-structured approach were significantly more unique (p=.003), and directionally more informative. These results influenced the final design of the prototype website.
Submission: CHI Conference Finalist
The final paper was a finalist in the Student Design Competition of CHI 2013. Check out the paper here. Also, see the supplementary materials here for more information about the design and my research.