Joel Chan | Papers

Here is a (mostly complete, but probably not fully up to date) list of all my peer-reviewed publications. A more comprehensive and up-to-date (but slightly noisy) listing is available at my Google Scholar profile.

2023

The Importance of Separating Appropriateness into Impact and Feasibility for the Psychology of Creativity Chan, Joel, and Schunn, Christian D. Creativity Research Journal 2023 [Abstract]
Scientific progress on creativity research depends on having properly operationalized measures. In psychological research on creativity, it is common to operationalize creativity as the combination of novelty and appropriateness. However, the operationalization of appropriateness varies widely across researchers, studies, and domains (e.g. technical goodness, significance, elegance, usefulness, and feasibility). We argue that a core distinction between impact (how useful an idea is for solving the problem) and feasibility (how easy it is to realize the idea) underlies the variation. We further claim that this distinction is both possible to capture reliably in practice and psychologically significant. To test these claims, 318 ideas from 5 real-world social innovation problems (e.g. improving accessibility in elections) were rated for novelty, impact, and feasibility by a set of six experts selected for each of the 5 challenges. We find that all three constructs can be measured reliably and are statistically separable. Further, we show that distinguishing impact and feasibility reveals theoretically meaningful patterns of relationships with key psychological processes of creativity – analogy and conceptual combination – that would be difficult if impact and feasibility were conflated. These results demonstrate the theoretical importance of separating appropriateness into impact and feasibility for the psychology of creativity.
Fluid Transformers and Creative Analogies: Exploring Large Language Models’ Capacity for Augmenting Cross-Domain Analogical Creativity Ding, Zijian, Srinivasan, Arvind, Macneil, Stephen, and Chan, Joel In Proceedings of the 15th Conference on Creativity and Cognition 2023 Best Paper Honorable Mention [Abstract] [PDF]
Cross-domain analogical reasoning is a core creative ability that can be challenging for humans. Recent work has shown some proofs-of-concept of Large language Models’ (LLMs) ability to generate cross-domain analogies. However, the reliability and potential usefulness of this capacity for augmenting human creative work has received little systematic exploration. In this paper, we systematically explore LLMs capacity to augment cross-domain analogical reasoning. Across three studies, we found: 1) LLM-generated cross-domain analogies were frequently judged as helpful in the context of a problem reformulation task (median 4 out of 5 helpfulness rating), and frequently (∼ 80% of cases) led to observable changes in problem formulations, and 2) there was an upper bound of ∼ 25% of outputs being rated as potentially harmful, with a majority due to potentially upsetting content, rather than biased or toxic content. These results demonstrate the potential utility — and risks — of LLMs for augmenting cross-domain analogical creativity.
Exploring Challenges to Inclusion in Participatory Design From the Perspectives of Global North Practitioners Elsayed-Ali, Salma, Bonsignore, Elizabeth, and Chan, Joel Proceedings of the ACM on Human-Computer Interaction 2023 [Abstract] [PDF]
Participatory Design (PD) aims to promote inclusivity by involving users throughout the design process. However, Human-Computer Interaction (HCI) and social computing research have pointed to instances where PD as practiced can, paradoxically, be exclusive. We aim to understand some of the challenges that could lead to exclusivity in order to design more inclusive PD practices. To investigate this, we conducted interviews with ten expert PD practitioners based in the Global North whose focus is on inclusion. Synthesizing practitioners’ accounts, we advance understandings of challenges surrounding: 1) instantiating shared spaces that empower partners; 2) developing common ground among stakeholders; and 3) balancing funding needs with open-ended PD. We contribute theoretical and empirical insights into these challenges and close by articulating potential implications for addressing these challenges to inclusion in PD.
CausalMapper: Challenging designers to think in systems with Causal Maps and Large Language Model Huang, Ziheng, Quan, Kexin, Chan, Joel, and MacNeil, Stephen In Proceedings of the 15th Conference on Creativity and Cognition 2023 [Abstract]
Professional designers often construct and explore conceptual representations (e.g.: design spaces) to help them reason about complex design situations and consider potential design pitfalls. However, it is often challenging, even for professional designers, to exhaustively consider the many pitfalls that might result from design activity. We present CausalMapper, a mixed-initiative system, that leverages a large language model (LLM) and a causal map representation to teach design students how to reason about the relationships between problems and solutions. Where creativity support tools often focus on ideating creative solutions, our mixed-initiative approach focuses on ideating ecosystems of solutions that holistically address a set of related problems. By leveraging the generative creativity of LLMs, designers are inspired to consider solutions and potential consequences that emerge when solutions are adopted. At the same time, leveraging the designers’ domain knowledge to account for and correct the biases inherent in LLMs. Through a case study, we demonstrate the functionality of this mixed-initiative system. The goal of this demo is to present a creativity support tool that is intended to teach design students to think more systematically by generating ideas that challenge their thinking rather just augmenting their creative potential.
AnalogiLead: Improving Selection of Analogical Inspirations with Chunking and Recombination Srinivasan, Arvind, and Chan, Joel In Proceedings of the 15th Conference on Creativity and Cognition 2023 [Abstract] [PDF]
Analogical reasoning, a process that integrates potential leads across domains and disciplines, has been proven to contribute to breakthrough innovations. Selecting the right analogical leads is crucial, as it determines the quality and effectiveness of the generated ideas. However, identifying relevant analogical leads can be challenging and may be missed due to premature rejection or design fixation. To address this problem, our system, "AnalogiLead", draws on the cognitive mechanisms of chunking and recombination as a medium of interaction for selecting beneficial analogies. Users interact with meaningful chunks or segments from a design problem and analogy, represented as interactive tiles called "magnets", and evaluate the analogies by recombining the "magnets" into brainstorming questions. These mechanisms are designed to foster playful and divergent exploration of analogical leads (vs. restrictive, relevance-based screening), to reduce premature rejection of analogical leads and foster more analogical innovations.
How Diverse Initial Samples Help and Hurt Bayesian Optimizers Kamrah, Eesh, Ghoreishi, Fatemeh, Ding, Zijian, Chan, Joel, and Fuge, Mark Journal of Mechanical Design 2023 [Abstract]
Design researchers have struggled to produce quantitative predictions for exactly why and when diversity might help or hinder design search efforts. This paper addresses that problem by studying one ubiquitously used search strategy – Bayesian Optimization (BO) – on a 2D test problem with modifiable difficulty. Specifically, we test how providing diverse versus non-diverse initial samples to BO affects its performance during search and introduce a fast DPP sampling method for computing diverse sets to detect sets of highly diverse or non-diverse initial samples. We initially found, to our surprise, that diversity did not affect BO, neither helping nor hurting its convergence. However, follow-on experiments illuminated a trade-off. Non-diverse initial samples hastened posterior convergence for the underlying model hyper-parameters – a Model Building advantage. In contrast, diverse initial samples accelerated exploring the function itself – a Space Exploration advantage. Both advantages help BO, but in different ways, and the initial sample diversity modulates how BO trades those advantages. We show that fixing the BO hyper-parameters removes the Model Building advantage, causing diverse initial samples to always outperform models trained with non-diverse samples. These findings shed light on why, at least for BO-type optimizers, the use of diversity has mixed effects and cautions against the ubiquitous use of space-filling initializations in BO. To the extent that humans use explore-exploit search strategies similar to BO, our results provide a testable conjecture for why and when diversity may affect human-subject or design team experiments.
Supporting Exploration of Far-Domain Analogical Inspirations with Bridging Analogies Chan, Joel, and Rudd, David Anthony In Proceedings of the 15th Conference on Creativity and Cognition 2023 [Abstract]
Far domain analogies — relational comparisons between topics that seem very different on the surface, such as the solar system and an atom — can be a powerful source of new ideas. However, people struggle to benefit from them. We explore how bridging analogies, analogies that bridge between a knowledge anchor that is familiar to the problem solver, and a target analogy, can help innovators to benefit from far domain analogies. Utilizing a breadth first search for shortest paths between concepts on Wikipedia, we identified potential bridging analogies that connect a participant’s knowledge anchor to a far domain analogy. In a think aloud study, participants brainstormed on three design challenges, with only far analogies, or far analogies and bridging analogies tailored to their knowledge anchors. We observed that bridging analogies aided participants in producing more abstract solutions vs. more direct translation of the analogies in their solution.

2022

Augmenting Scientific Creativity with an Analogical Search Engine Kang, Hyeonsu B., Qian, Xin, Hope, Tom, Shahaf, Dafna, Chan, Joel, and Kittur, Aniket ACM Transactions on Computer-Human Interaction 2022 [Abstract] [PDF]
Analogies have been central to creative problem-solving throughout the history of science and technology. As the number of scientific papers continues to increase exponentially, there is a growing opportunity for finding diverse solutions to existing problems. However, realizing this potential requires the development of a means for searching through a large corpus that goes beyond surface matches and simple keywords. Here we contribute the first end-to-end system for analogical search on scientific papers and evaluate its effectiveness with scientists’ own problems. Using a human-in-the-loop AI system as a probe we find that our system facilitates creative ideation, and that ideation success is mediated by an intermediate level of matching on the problem abstraction (i.e., high versus low). We also demonstrate a fully automated AI search engine that achieves a similar accuracy with the human-in-the-loop system. We conclude with design implications for enabling automated analogical inspiration engines to accelerate scientific innovation.
Growing New Scholarly Communication Infrastructures for Sharing, Reusing, and Synthesizing Knowledge Chan, Joel, Lutters, Wayne, Schneider, Jodi, Kirsanow, Karola, Bessa, Silvia, and Saunders, Jonny L. In Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing 2022 [Abstract] [PDF]
Sharing, reuse, and synthesis of knowledge is central to the research process. These core functions are in theory served by the system of monographs, abstracts, and papers in journals and proceedings, with citation indices and search databases that comprise the core of our formal scholarly communication infrastructure; yet, converging lines of empirical and anecdotal evidence suggest that this system does not adequately act as infrastructure for synthesis. Emerging developments in new institutions for science, along with new technical infrastructures and tooling for decentralized knowledge work, offer new opportunities to prototype new technical infrastructures on top of a different installed base than the publish or perish, neoliberal academy. This workshop aims to integrate these developments and communities with CSCW’s deep roots in knowledge infrastructures and collaborative and distributed sensemaking, with new developments in science institutions and tooling, to stimulate and accelerate progress towards prototyping new scholarly communication infrastructures that are actually optimized for sharing, reusing, and synthesizing knowledge.
Scaling Creative Inspiration with Fine-Grained Functional Aspects of Ideas Hope, Tom, Tamari, Ronen, Hershcovich, Daniel, Kang, Hyeonsu B, Chan, Joel, Kittur, Aniket, and Shahaf, Dafna In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems 2022 [Abstract] [PDF]
Large repositories of products, patents and scientific papers offer an opportunity for building systems that scour millions of ideas and help users discover inspirations. However, idea descriptions are typically in the form of unstructured text, lacking key structure that is required for supporting creative innovation interactions. Prior work has explored idea representations that were either limited in expressivity, required significant manual effort from users, or dependent on curated knowledge bases with poor coverage. We explore a novel representation that automatically breaks up products into fine-grained functional aspects capturing the purposes and mechanisms of ideas, and use it to support important creative innovation interactions: functional search for ideas, and exploration of the design space around a focal problem by viewing related problem perspectives pooled from across many products. In user studies, our approach boosts the quality of creative search and inspirations, substantially outperforming strong baselines by 50-60%.

2021

Sustainable Authorship Models for a Discourse-Based Scholarly Communication Infrastructure Chan, Joel Common Place 2021 [Abstract] [PDF]
Integrating process into scholarly practices of reading, writing, and knowledge production
Learning to Recommend Visualizations from Data Qian, Xin, Rossi, Ryan A., Du, Fan, Kim, Sungchul, Koh, Eunyee, Malik, Sana, Lee, Tak Yeon, and Chan, Joel In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining 2021 [Abstract] [PDF]
Visualization recommendation is important for exploratory analysis and making sense of the data quickly by automatically recommending relevant visualizations to the user. In this work, we propose the first end-to-end ML-based visualization recommendation system that leverages a large corpus of datasets and their relevant visualizations to learn a visualization recommendation model automatically. Then, given a new unseen dataset from an arbitrary user, the model automatically generates visualizations for that new dataset, derives scores for the visualizations, and outputs a list of recommended visualizations to the user ordered by effectiveness. We also describe an evaluation framework to quantitatively evaluate visualization recommendation models learned from a large corpus of visualizations and datasets. Through quantitative experiments, a user study, and qualitative analysis, we show that our end-to-end ML-based system recommends more effective and useful visualizations compared to existing state-of-the-art rule-based systems.
Generating Accurate Caption Units for Figure Captioning Qian, Xin, Koh, Eunyee, Du, Fan, Kim, Sungchul, Chan, Joel, Rossi, Ryan A., Malik, Sana, and Lee, Tak Yeon In Proceedings of the Web Conference 2021 2021 [Abstract] [PDF]
Scientific-style figures are commonly used on the web to present numerical information. Captions that tell accurate figure information and sound natural would significantly improve figure accessibility. In this paper, we present promising results on machine figure captioning. A recent corpus analysis of real-world captions reveals that machine figure captioning systems should start by generating accurate caption units. We formulate the caption unit generation problem as a controlled captioning problem. Given a caption unit type as a control signal, a model generates an accurate caption unit of that type. As a proof-of-concept on single bar charts, we propose a model, FigJAM, that achieves this goal through utilizing metadata information and a joint static and dynamic dictionary. Quantitative evaluations with two datasets from the figure question answering task show that our model can generate more accurate caption units than competitive baseline models. A user study with ten human experts confirms the value of machine-generated caption units in their standalone accuracy and naturalness. Finally, a post-editing simulation study demonstrates the potential for models to paraphrase and stitch together single-type caption units into multi-type captions by learning from data.
Managing Context during Scholarly Knowledge Synthesis: Process Patterns and System Mechanics Morabito, John S, and Chan, Joel In Creativity and Cognition 2021 [Abstract] [PDF]
Scholarly knowledge synthesis — the production of a novel conceptual whole such as an effective literature review or theory — is a critical yet consistently challenging subtask of research. We explore how managing the context of knowledge claims being synthesized, such as their production context or methodology, may be a critical under-supported subtask of synthesis in existing tools. Through in situ protocol analyses of researchers doing the work of synthesis, we studied how researchers capture contextual information in their notes and annotations, and how this varies across generic vs. specialized systems for synthesis. Our analysis revealed common process patterns of context capture, and qualitative differences in the nature of support for context capture across generic and specialized systems. Based on these findings, we discuss design implications for systems that aim to better support scholarly synthesis.
Supporting human flourishing by ensuring human involvement in AI-infused systems Chan, Joel, Daumé III, Hal, Dickerson, John P., Kacorri, Hernisa, and Shneiderman, Ben In NeurIPS 2021 Workshop on Human Centered AI 2021

2020

Opening Up the Black Box of Scholarly Synthesis: Intermediate Products, Processes, and Tools. Qian, Xin, Fenlon, Katrina, Lutters, Wayne G., and Chan, Joel In Proceedings of ASIST 2020 2020
Understanding Older Adults’ Participation in Design Workshops Pradhan, Alisha, Jelen, Ben, Siek, Katie A., Chan, Joel, and Lazar, Amanda In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems 2020 [Abstract]
Design workshops are a popular means of including older adults in technology development. However, there are open questions around how to best scaffold this participation, particularly in supporting older adults to associate their designs with themselves, rather than designing for an “other older adult.” By conducting workshops focusing on envisioning the future of internet of things (IoT) technologies at home, we provide an understanding of how older individuals participate in group activities to conceptualize technology for themselves. We find that at different stages of the design process, individuals shift in who they envision the end user of the technology: at first, they think about common older adult needs, then turn to designing for themselves. Individuals’ attitudes towards technology also impact group dynamics along with final design ideas. Our discussion contributes to an understanding of how to support older adults in designing for themselves, new perspectives on aging-in-place technologies, and recommendations for configuring design workshops with older individuals.
A Formative Study on Designing Accurate and Natural Figure Captioning Systems Qian, Xin, Koh, Eunyee, Du, Fan, Kim, Sungchul, and Chan, Joel In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems 2020 [Abstract]
Automatic figure captioning is widely useful for improving the readability and accessibility of figures. Despite recent advances in figure question answering and parsing figure elements that enable machines to accurately read information from figures, the machine learning community still lacks sufficient understanding of this problem, on what contents are important to include in a caption and how to make it sound natural. In this work, we crawled, annotated, and analyzed a corpus of real-world human-written figure captions. Our study results show that real-world captions usually consist of a finite set of caption units and that automatic figure captioning should be formulated as a multi-stage task. The first stage is to generate caption units with high accuracy and the second is to stitch together the units with diverse stitching patterns, to form a natural caption.
Where the rubber meets the road: Identifying integration points for semantic publishing in existing scholarly practice Chan, Joel, Qian, Xin, Fenlon, Katrina, and Lutters, Wayne G. In JCDL 2020 Workshop on Conceptual MOdeling 2020 [Abstract]
Semantic publishing has significant potential to transform scholarly work. While much progress has been made on conceptual models and technical infrastructure, authorship remains an open problem. Here, we explore whether and how semantic publishing labor might be integrated into the existing practices of scholars reading and synthesizing the literature. From a series of studies of scholars, we observe rich practices across a variety of workflows and tool ecologies that overlap with key aspects of semantic publishing: 1) creating granular knowledge artifacts (}rcompress{}), extracting and specifying provenance and contextual details (}rcontext{}), and 3) specifying semantically typed entities and relations between knowledge units (}rcompose{}). We discuss implications of these findings for developing sustainable, scholar-powered models of semantic publishing.

2019

Scaling up Analogical Innovation with Crowds and AI Kittur, Aniket, Yu, Lixiu, Hope, Tom, Chan, Joel, Lifshitz-Assaf, Hila, Gilon, Karni, Ng, Felicia, Kraut, Robert E., and Shahaf, Dafna Proceedings of the National Academy of Sciences 2019 [Abstract] [PDF]
Analogy—the ability to find and apply deep structural patterns across domains—has been fundamental to human innovation in science and technology. Today there is a growing opportunity to accelerate innovation by moving analogy out of a single person’s mind and distributing it across many information processors, both human and machine. Doing so has the potential to overcome cognitive fixation, scale to large idea repositories, and support complex problems with multiple constraints. Here we lay out a perspective on the future of scalable analogical innovation and first steps using crowds and artificial intelligence (AI) to augment creativity that quantitatively demonstrate the promise of the approach, as well as core challenges critical to realizing this vision.
Beyond iTunes for Papers: Redefining the Unit of Interaction in Literature Review Tools Qian, Xin, Erhart, Matt J., Kittur, Aniket, Lutters, Wayne G., and Chan, Joel In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing 2019 [Abstract]
Conducting an effective literature review is an essential step in all scientific work. However, the process is difficult, particularly for interdisciplinary work. Here, we articulate a key avenue for improvement for literature review tools: supporting the appropriate unit of interaction, which we argue is a "grounded claim", a concise statement linked to key contextual details such as evidence. However, there are significant cognitive and interaction costs in creating them. We share insights from our development of a prototype literature review tool, the Knowledge Compressor, that aims to lower these costs.
How science teams deal with uncertainty Chan, Joel, and Paletz, Susannah BF The Oxford Handbook of Group Creativity and Innovation 2019

2018

Analogy Mining for Specific Design Needs Gilon, Karni, Chan, Joel, Ng, Felicia Y, Assaf, Hila Lifshitz, Kittur, Aniket, and Shahaf, Dafna In Proceedings of the 2018 ACM SIGCHI Conference on Human Factors in Computing 2018 [Abstract] [PDF]
Finding analogical inspirations in distant domains is a powerful way of solving problems. However, as the number of inspirations that could be matched and the dimensions on which that matching could occur grow, it becomes challenging for designers to find inspirations relevant to their needs. Furthermore, designers are often interested in exploring specific aspects of a product– for example, one designer might be interested in improving the brewing capability of an outdoor coffee maker, while another might wish to optimize for portability. In this paper we introduce a novel system for targeting analogical search for specific needs. Specifically, we contribute an analogical search engine for expressing and abstracting specific design needs that returns more distant yet relevant inspirations than alternate approaches.
SOLVENT: A Mixed Initiative System for Finding Analogies between Research Papers Chan, Joel, Chang, Joseph Chee, Hope, Tom, Shahaf, Dafna, and Kittur, Aniket Proceedings of ACM Human-Computer Interaction: CSCW 2018 [Abstract] [PDF]
Scientific discoveries are often driven by finding analogies in distant domains, but the growing number of papers makes it difficult to find relevant ideas in a single discipline, let alone distant analogies in other domains. To provide computational support for finding analogies across domains, we introduce SOLVENT, a mixed-initiative system where humans annotate aspects of research papers that denote their background (the high-level problems being addressed), purpose (the specific problems being addressed), mechanism (how they achieved their purpose), and findings (what they learned/achieved), and a computational model constructs a semantic representation from these annotations that can be used to find analogies among the research papers. We demonstrate that this system finds more analogies than baseline information-retrieval approaches; that annotators and annotations can generalize beyond domain; and that the resulting analogies found are useful to experts. These results demonstrate a novel path towards computationally supported knowledge sharing in research communities

2017

Accelerating Innovation Through Analogy Mining Hope, Tom, Chan, Joel, Kittur, Aniket, and Shahaf, Dafna In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2017 Best Paper [Abstract] [PDF]
The availability of large idea repositories (e.g., the U.S. patent database) could significantly accelerate innovation and discovery by providing people with inspiration from solutions to analogous problems. However, finding useful analogies in these large, messy, real-world repositories remains a persistent challenge for either human or automated methods. Previous approaches include costly hand-created databases that have high relational structure (e.g., predicate calculus representations) but are very sparse. Simpler machine-learning/information-retrieval similarity metrics can scale to large, natural-language datasets, but struggle to account for structural similarity, which is central to analogy. In this paper we explore the viability and value of learning simpler structural representations, specifically, “problem schemas”, which specify the purpose of a product and the mechanisms by which it achieves that purpose. Our approach combines crowdsourcing and recurrent neural networks to extract purpose and mechanism vector representations from product descriptions. We demonstrate that these learned vectors allow us to find analogies with higher precision and recall than traditional information-retrieval methods. In an ideation experiment, analogies retrieved by our models significantly increased people’s likelihood of generating creative ideas compared to analogies retrieved by traditional methods. Our results suggest a promising approach to enabling computational analogy at scale is to learn and leverage weaker structural representations.
The dynamics of micro-conflicts and uncertainty in successful and unsuccessful design teams Paletz, Susannah B. F., Chan, Joel, and Schunn, Christian D. Design Studies 2017 [Abstract] [PDF]
What differentiates successful from unsuccessful design teams? Building on new research on design innovation that emphasizes interactions between social and cognitive processes, we investigated a potential distinguishing feature: Successful design teams may harness interpersonal conflicts (a social design process) to mitigate uncertainty (a cognitive design process). We analyzed temporal relationships between brief, expressed interpersonal disagreements and subsequent spoken individual uncertainty in 30 h of conversations of 10 successful and 11 unsuccessful engineering product design teams. We discovered that micro-conflicts were followed by a relative reduction in uncertainty in successful design teams, whereas uncertainty rose after micro-conflicts in unsuccessful design teams, suggesting that interactions between conflict and uncertainty may be a differentiating factor for design team success.
Crowdboard: Augmenting In-Person Idea Generation with Real-Time Crowds Andolina, Salvatore, Schneider, Hendrik, Chan, Joel, Klouche, Khalil, Jacucci, Giulio, and Dow, Steven In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition 2017 [Abstract] [PDF]
Online crowds can help infuse creativity into the design process, but traditional strategies for leveraging them, such as large-scale ideation platforms, require time and organizational effort in order to obtain results. We propose a new method for crowd-based ideation that simplifies the process by having smaller crowds join in-person ideators during synchronous creative sessions. Our system Crowdboard allows online crowds to provide real-time creative input during early-stage design activities, such as brainstorming or concept mapping. The system enables in-person ideators to develop ideas on a physical or digital whiteboard which is augmented with real-time creative input from online participants who see and hear a live broadcast of the meeting. We validate Crowdboard via two user studies in which dyads of in-person ideators brainstormed with the help of crowd ideators. Our studies suggest that Crowdboard can effectively enhance ongoing brainstorming sessions, but also revealed key challenges for how to better facilitate interactions among in-person and crowd ideators.
Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation Chan, Joel, Siangliulue, Pao, Qori McDonald, Denisa, Liu, Ruixue, Moradinezhad, Reza, Aman, Safa, Solovey, Erin T., Gajos, Krzysztof Z., and Dow, Steven P. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition 2017 [Abstract] [PDF]
Collaborative ideation systems can help people generate more creative ideas by exposing them to ideas different from their own. However, there are competing theoretical views on whether and when such exposure is helpful. Associationist theory suggests that exposing ideators to ideas that are semantically far from their own maximizes novel combinations of ideas. In contrast, SIAM theory cautions that systems should offer far ideas only when ideators reach an impasse (a cognitive state in which they have exhausted ideas within a particular category), and offer near ideas during productive ideation (a cognitive state in which they are actively exploring ideas within a category), which maximizes exploration within categories. Our research compares these theoretical recommendations. In an online experiment, 245 participants generated ideas for a themed wedding; we detected and validated participants’ cognitive states using a combination of behavioral and neuroimaging data. Receiving far ideas during productive ideation resulted in slower ideation and less within-category exploration, without significant benefits for novelty, compared to receiving no inspirations. Participants were also more likely to hit an impasse when receiving far ideas during productive ideation. These findings suggest that far inspirational ideas can harm creativity if received during productive ideation.
Measuring design innovation for project-based design assessment: considerations of robustness and efficiency Schunn, Christian, Chan, Joel, and Goncher, Andrea Bitácora Urbano Territorial 2017 [Abstract] [PDF]
Instructional approaches that support the acquisition of innovation design process skills for engineering and other design students are critical to developing design competencies. Resources that enable efficient and valid evaluation of design outcomes are needed; however current evaluation methods do not apply well to the heterogeneous projects found in authentic project-based design classes such as capstone design. We develop a robust and efficient measure of design outcome innovation and validate our measure with a large and diverse set of design outcomes from a project based design class. The measure is based on expert judgments of designconcepts’ value and functionality derived from a set of importance-weighted design requirements. In the context of an engineering design class, the innovation score was a statistically significant predictor of success in terms of implementation status by the client. Thus, the measure’s validity was supported by its ability to predict design concepts’ implementation by clients in the context of a product realization class. New design outcome assessment measures in the context of authentic project-based design environments, such as the one developed in the present study, should interface with process-based metrics to create higher-quality input into the overall assessment of design team performance.

2016

Scaling up Analogy with Crowdsourcing and Machine Learning. Chan, Joel, Hope, Tom, Shahaf, Dafna, and Kittur, Aniket In ICCBR Workshops 2016 [Abstract] [PDF]
Despite tremendous advances in computational models of human analogy, a persistent challenge has been scaling up to find useful analogies in large, messy, real-world data. The availability of large idea repositories (e.g., the U.S. patent database) could significantly accelerate innovation and discovery in a way never previously possible. Previous approaches have been limited by relying on hand-created databases that have high relational structure but are very sparse (e.g., predicate calculus representations). Traditional machine-learning/information-retrieval similarity metrics (e.g., LSA) can scale to large, natural-language datasets; however, while these methods are good at detecting surface similarity, they struggle to account for structural similarity. In this paper, we pro- pose to leverage crowdsourcing techniques to construct a dataset with rich “analogy-tuning” signals, used to guide machine learning models towards matches based on relations rather than surface features. We demonstrate our approach with a crowdsourced analogy identification task, whose results are used to train deep learning algorithms. Our initial results suggest that a deep learning model trained on positive/negative example analogies from the task can find more analogous matches than an LSA baseline, and that incorporating behavioral signals (such as queries used to retrieve an analogy) can further boost its performance.
Situative Creativity: Larger Physical Spaces Facilitate Thinking of Novel Uses for Everyday Objects Chan, Joel, and Nokes-Malach, Timothy The Journal of Problem Solving 2016 [Abstract] [PDF]
People often use spatial metaphors (e.g., think “laterally,” “outside the box”) to describe exploration of the problem space during creative problem solving. In this paper, we probe the potential cognitive underpinnings of these spatial metaphors. Drawing on theories of situative cognition, semantic foraging theory, and environmental psychology, we formulate and test the hypothesis that larger physical spaces can facilitate divergent (but not convergent) processes in problem space exploration. Across two experiments, participants worked on a battery of problem solving tasks intended to represent divergent (alternative uses, shape invention) and convergent (remote associates, letter extrapolation) problem solving processes in either a large or a small room. In Experiment 1, participants in the larger room produced more novel alternative uses for everyday objects, and created more novel shape inventions, but generated less practical alternative uses, than participants in the smaller room. In Experiment 2, participants in the larger room (including a variant larger room) also produced more novel alternative uses for everyday objects, and less practical alternative uses, than participants in a small room, but did not create more novel shape inventions. These results suggest that spatial metaphors for problem space exploration may reflect meaningful cognitive phenomena: People may be able to search more broadly in a problem space if they are in an environment where broad physical search is a salient affordance; however, this effect appears to be relatively small and may depend on having sufficiently motivated participants.
Comparing Different Sensemaking Approaches for Large-Scale Ideation Chan, Joel, Dang, Steven, and Dow, Steven P. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems 2016 [Abstract] [PDF]
arge-scale idea generation platforms often expose ideators to previous ideas. However, research suggests people generate better ideas if they see abstracted solution paths (e.g., descriptions of solution approaches generated through human sensemaking) rather than being inundated with all prior ideas. Automated and semi-automated methods can also offer interpretations of earlier ideas. To benefit from sensemaking in practice with limited resources, ideation platform developers need to weigh the cost-quality tradeoffs of different methods for surfacing solution paths. To explore this, we conducted an online study where 245 participants generated ideas for two problems in one of five conditions: 1) no stimuli, 2) exposure to all prior ideas, or solution paths extracted from prior ideas using 3) a fully automated workflow, 4) a hybrid human-machine approach, and 5) a fully manual approach. Contrary to expectations, human-generated paths did not improve ideation (as meas- ured by fluency and breadth of ideation) over simply showing all ideas. Machine-generated paths sometimes significantly improved fluency and breadth of ideation over no ideas (although at some cost to idea quality). These findings suggest that automated sensemaking can improve idea generation, but we need more research to understand the value of human sensemaking for crowd ideation.
Uncovering Uncertainty through Disagreement Paletz, Susannah B. F., Chan, Joel, and Schunn, Christian D. Applied Cognitive Psychology 2016 [Abstract] [PDF]
This study explored the association between different types of brief disagreements and subsequent levels of expressed psychological uncertainty, a fundamental cognitive aspect of complex problem solving. We examined 11 hours (11 861 utterances) of conversations in expert science teams, sampled across the first 90 days of the Mars Exploration Rover mission. Utterances were independently coded for micro-conflicts and expressed psychological uncertainty. Using time-lagged hierarchical linear modeling applied to blocks of 25 utterances, we found that micro-conflicts regarding rover planning were followed by greater uncertainty. Brief disagreements about science issues were followed by an increase in expressed uncertainty early in the mission. Examining the potential reverse temporal association, uncertainty actually predicted fewer subsequent disagreements, ruling out indirect, third variable associations of conflict and uncertainty. Overall, these findings suggest that some forms of disagreement may serve to uncover important areas of uncertainty in complex teamwork, perhaps via revealing differences in mental models.Copyright © 2016 John Wiley & Sons, Ltd.
IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-Powered Real-time Semantic Modeling Siangliulue, Pao, Chan, Joel, Dow, Steven P., and Gajos, Krzysztof Z. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology 2016 [Abstract] [PDF]
Prior work on creativity support tools demonstrates how a computational semantic model of a solution space can enable interventions that substantially improve the number, quality and diversity of ideas. However, automated semantic modeling often falls short when people contribute short text snippets or sketches. Innovation platforms can employ humans to provide semantic judgments to construct a semantic model, but this relies on external workers completing a large number of tedious micro tasks. This requirement threatens both accuracy (external workers may lack expertise and context to make accurate semantic judgments) and scalability (external workers are costly). In this paper, we introduce IdeaHound, an ideation system that seamlessly integrates the task of defining semantic relationships among ideas into the primary task of idea generation. The system combines implicit human actions with machine learning to create a computational semantic model of the emerging solution space. The integrated nature of these judgments allows IDEAHOUND to leverage the expertise and efforts of participants who are already motivated to contribute to idea generation, overcoming the issues of scalability inherent to existing approaches. Our results show that participants were equally willing to use (and just as productive using) IDEAHOUND compared to a conventional platform that did not require organizing ideas. Our integrated crowdsourcing approach also creates a more accurate semantic model than an existing crowdsourced approach (performed by external crowds). We demonstrate how this model enables helpful creative interventions: providing diverse inspirational examples, providing similar ideas for a given idea and providing a visual overview of the solution space.
Improving Crowd Innovation with Expert Facilitation Chan, Joel, Dang, Steven, and Dow, Steven P. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing 2016 [Abstract] [PDF]
Online crowds are a promising source of new innovations. However, crowd innovation quality does not always match its quantity. In this paper, we explore how to improve crowd innovation with real-time expert guidance. One approach would for experts to provide personalized feed-back, but this scales poorly, and may lead to premature convergence during creative work. Drawing on strategies for facilitating face-to-face brainstorms, we introduce a crowd ideation system where experts monitor incoming ideas through a dashboard and offer high-level “inspirations” to guide ideation. A series of controlled experiments show that experienced facilitators increased the quantity and creativity of workersâ ideas compared to unfacilitated workers, while Novice facilitators reduced workersâ creativity. Analyses of inspiration strategies suggest these opposing results stem from differential use of successful inspiration strategies (e.g., provoking mental simulations). The results show that expert facilitation can significantly improve crowd innovation, but inexperienced facilitators may need scaffolding to be successful.

2015

Providing timely examples improves the quantity and quality of generated ideas Siangliulue, Pao, Chan, Joel, Gajos, Krzysztof, and Dow, Steven P. In Proceedings of the ACM Conference on Creativity and Cognition 2015 Best Contribution to Creative Communication (nominated) [Abstract] [PDF]
Emerging online ideation platforms with thousands of example ideas provide an important resource for creative production. But how can ideators best use these examples to create new innovations? Recent work has suggested that not just the choice of examples, but also the timing of their delivery can impact creative outcomes. Building on existing cognitive theories of creative insight, we hypothesize that people are likely to benefit from examples when they run out of ideas. We explore two example delivery mechanisms that test this hypothesis: 1) a system that proactively provides examples when a user appears to have run out of ideas, and 2) a system that provides examples when a user explicitly requests them. Our online experiment (N=97) compared these two mecha- nisms against two baselines: providing no examples and au- tomatically showing examples at a regular interval. Participants who requested examples themselves generated ideas that were rated the most novel by external evaluators. Partic- ipants who received ideas automatically when they appeared to be stuck produced the most ideas. Importantly, participants who received examples at a regular interval generated fewer ideas than participants who received no examples, suggesting that mere access to examples is not sufficient for creative inspiration. These results emphasize the importance of the timing of example delivery. Insights from this study can in- form the design of collective ideation support systems that help people generate many high quality ideas.
Do The Best Design Ideas (Really) Come From Conceptually Distant Sources Of Inspiration? Chan, Joel, Dow, Steven P., and Schunn, Christian D. Design Studies 2015 Design Studies Award [Abstract] [PDF]
Design ideas often come from sources of inspiration (e.g., analogous designs, prior experiences). In this paper, we test the popular but unevenly supported hypothesis that conceptually distant sources of inspiration provide the best insights for creative production. Through text analysis of hundreds of design concepts across a dozen different design challenges on a Web-based innovation platform that tracks connections to sources of inspiration, we find that citing sources is associated with greater creativity of ideas, but conceptually closer rather than farther sources appear more beneficial. This inverse relationship between conceptual distance and design creativity is robust across different design problems on the platform. In light of these findings, we revisit theories of design inspiration and creative cognition.
The impact of analogies on creative concept generation: Lessons from an in vivo study in engineering design Chan, Joel, and Schunn, Christian D. Cognitive Science 2015 [Abstract] [PDF]
Research on innovation often highlights analogies from sources outside the current problem domain as a major source of novel concepts; however, the mechanisms underlying this relationship are not well understood. We analyzed the temporal interplay between far analogy use and creative concept generation in a professional design teams brainstorming conversations, investigating the hypothesis that far analogies lead directly to very novel concepts via large steps in conceptual spaces (jumps). Surprisingly, we found that concepts were more similar to their preceding concepts after far analogy use compared to baseline situations (i.e., without far analogy use). Yet, far analogies increased the teams concept generation rate compared to baseline conditions. Overall, these results challenge the view that far analogies primarily lead to novel concepts via jumps in conceptual spaces, and suggest alternative pathways from far analogies to novel concepts (e.g., iterative, deep exploration within a functional space).
The importance of iteration in creative conceptual combination Chan, Joel, and Schunn, Christian D. Cognition 2015 [Abstract] [PDF]
Theories of creative conceptual combination hypothesize that, to generate highly creative concepts, one should attempt to combine source concepts that are very different from each other. While lab studies show a robust link between far combinations and increased novelty of concepts, empirical evidence that far combinations lead to more creative concepts (i.e., both more novel and of higher quality) is mixed. Drawing on models of the creative process, we frame conceptual combination as a divergent process, and hypothesize that iteration is necessary to convert far combinations into creative concepts. We trace conceptual genealogies of many hundreds of concepts proposed for a dozen different problems on a large-scale Web-based innovation platform, and model the effects of combination distance on creative outcomes of concepts. The results are consistent with our predictions: (1) direct effects of far combinations have a mean zero effect, and (2) indirect effects of far combinations (i.e., building on concepts that themselves build on far combinations) have more consistently positive effects. This pattern of effects is robust across problems on the platform. These findings lend clarity to theories of creative conceptual combination, and highlight the importance of iteration for generating creative concepts.

2014

The impact of sources of inspiration on the genesis of creative ideas Chan, Joel 2014 Tim Post Award [Abstract] [PDF]
Innovation fundamentally begins with a good idea. But where do good ideas come from? Much research suggests that innovative breakthroughs are often inspired by past experience: things and ideas that one has interacted with in the world. However, the same experiences that can inspire innovation can sometimes constrain or harm innovation through focus on previously unsuccessful solutions. In this dissertation, I explore principles for guiding interactions with sources of inspiration (previous/other ideas) to maximize their benefits and minimize their pitfalls, focusing on the role of conceptual distance and diversity of sources. I analyze thousands of ideas for complex innovation challenges (e.g., increasing accessibility in elections, revitalizing struggling urban areas) posted to an online crowd-sourced innovation platform that required contributors to cite sources of ideas, tracing the impact of the distance and diversity of sources in ideas’ conceptual genealogies on their creative success (as judged by an expert panel). In this dissertation, I make three primary contributions to the literature. First, leveraging techniques from natural language processing and machine learning, I develop a validated computational methodology for studying conceptual distance and diversity with complex design concepts, which addresses significant issues of efficiency and scalability faced in prior work. Second, I challenge the widespread but unevenly supported notion that far sources provide the best insights for creative ideation; addressing key methodological issues in prior work (time scale, statistical power, and problem variation), I show that overreliance on far sources can harm ideation success, and that good ideas can often come from very near sources. Finally, I demonstrate the potential value of incorporating a temporal dimension into analyses of the impact of sources of inspiration: I find evidence of differential impacts of source distance and diversity (viz., increased problem variation for the effect of source distance, and a more robust positive effect of source diversity) when considering sources farther back in ideas’ conceptual genealogies.

2013

Reducing Annotation Effort on Unbalanced Corpus based on Cost Matrix. Luo, Wencan, Litman, Diane J, and Chan, Joel In HLT-NAACL 2013 [Abstract] [PDF]
Annotated corpora play a significant role in many NLP applications. However, annotation by humans is time-consuming and costly. In this paper, a high recall predictor based on a cost-sensitive learner is proposed as a method to semi-automate the annotation of unbalanced classes. We demonstrate the effectiveness of our approach in the context of one form of unbalanced task: annotation of transcribed human-human dialogues for presence/absence of uncertainty. In two data sets, our cost-matrix based method of uncertainty annotation achieved high levels of recall while maintaining acceptable levels of accuracy. The method is able to reduce human annotation effort by about 80% without a significant loss in data quality, as demonstrated by an extrinsic evaluation showing that results originally achieved using manually-obtained uncertainty annotations can be replicated using semi-automatically obtained uncertainty annotations.
Expert representation of design repository space: A comparison to and validation of algorithmic output Fu, Katherine, Chan, Joel, Schunn, Christian, Cagan, Jonathan, and Kotovsky, Kenneth Design Studies 2013 [Abstract] [PDF]
Development of design-by-analogy tools is a promising design innovation research avenue. Previously, a method for computationally structuring patent databases as a basis for an automated design-by-analogy tool was introduced. To demonstrate its strengths and weaknesses, a computationally-generated structure is compared to four expert designers’ mental models of the domain. Results indicate that, compared to experts, the computationally-generated structure is sensible in clustering of patents and organization of clusters. The computationally-generated structure represents a space in which experts can find common ground/consensus making it promising to be intuitive/accessible to broad cohorts of designers. The computational method offers a resource-efficient way of usefully conceptualizing the space that is sensible to expert designers, while maintaining an element of unexpected representation of the space.
The Meaning of Near and Far: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output Fu, Katherine, Chan, Joel, Cagan, Jonathan, Kotovsky, Kenneth, Schunn, Christian, and Wood, Kristin Journal of Mechanical Design 2013 [Abstract] [PDF]
This work lends insight into the meaning and impact of “near” and “far” analogies. A cognitive engineering design study is presented that examines the effect of the distance of analogical design stimuli on design solution generation, and places those findings in context of results from the literature. The work ultimately sheds new light on the impact of analogies in the design process and the significance of their distance from a design problem. In this work, the design repository from which analogical stimuli are chosen is the U.S. patent database, a natural choice, as it is one of the largest and easily accessed catalogued databases of inventions. The “near” and “far” analogical stimuli for this study were chosen based on a structure of patents, created using a combination of latent semantic analysis and a Bayesian based algorithm for discovering structural form, resulting in clusters of patents connected by their relative similarity. The findings of this engineering design study are juxtaposed with the findings of a previous study by the authors in design by analogy, which appear to be contradictory when viewed independently. However, by mapping the analogical stimuli used in the earlier work into similar structures along with the patents used in the current study, a relationship between all of the stimuli and their relative distance from the design problem is discovered. The results confirm that “near” and “far” are relative terms, and depend on the characteristics of the potential stimuli. Further, although the literature has shown that “far” analogical stimuli are more likely to lead to the generation of innovative solutions with novel characteristics, there is such a thing as too far. That is, if the stimuli are too distant, they then can become harmful to the design process. Importantly, as well, the data mapping approach to identify analogies works, and is able to impact the effectiveness of the design process. This work has implications not only in the area of finding inspirational designs to use for design by analogy processes in practice, but also for synthesis, or perhaps even unification, of future studies in the field of design by analogy.

2012

The Meaning of “Near” and “Far”: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output Fu, Katherine, Chan, Joel, Cagan, Jonathan, Kotovsky, Kenneth, Schunn, Christian, and Wood, Kristin In ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference 2012 Best Paper [Abstract] [PDF]
This work lends insight into the meaning and impact of “near” and “far” analogies. A cognitive engineering design study is presented that examines the effect of the distance of analogical design stimuli on design solution generation, and places those findings in context of results from the literature. The work ultimately sheds new light on the impact of analogies in the design process and the significance of their distance from a design problem. In this work, the design repository from which analogical stimuli are chosen is the U.S. patent database, a natural choice, as it is one of the largest and easily accessed catalogued databases of inventions. The “near” and “far” analogical stimuli for this study were chosen based on a structure of patents, created using a combination of Latent Semantic Analysis and a Bayesian based algorithm for discovering structural form, resulting in clusters of patents connected by their relative similarity. The findings of this engineering design study are contextualized with the findings of recent work in design by analogy, by mapping the analogical stimuli used in the earlier work into similar structures along with the patents used in the current study. Doing so allows the discovery of a relationship between all of the stimuli and their relative distance from the design problem. The results confirm that “near” and “far” are relative terms, and depend on the characteristics of the potential stimuli. Further, although the literature has shown that “far” analogical stimuli are more likely to lead to the generation innovative solutions with novel characteristics, there is such a thing as too far. That is, if the stimuli are too distant, they then can become harmful to the design process. Importantly, as well, the data mapping approach to identify analogies works, and is able to impact the effectiveness of the design process. This work has implications not only in the area of finding inspirational designs to use for design by analogy processes in practice, but also for synthesis, or perhaps even unification, of future studies in the field of design by analogy.
Analogy as a strategy for supporting complex problem solving under uncertainty Chan, Joel, Paletz, Susannah BF, and Schunn, Christian D Memory & cognition 2012 [Abstract] [PDF]
Complex problem solving in naturalistic environments is fraught with uncertainty, which has significant impacts on problem-solving behavior. Thus, theories of human problem solving should include accounts of the cognitive strategies people bring to bear to deal with uncertainty during problem solving. In this article, we present evidence that analogy is one such strategy. Using statistical analyses of the temporal dynamics between analogy and expressed uncertainty in the naturalistic problem-solving conversations among scientists on the Mars Rover Mission, we show that spikes in expressed uncertainty reliably predict analogy use (Study 1) and that expressed uncertainty reduces to baseline levels following analogy use (Study 2). In addition, in Study 3, we show with qualitative analyses that this relationship between uncertainty and analogy is not due to miscommunication-related uncertainty but, rather, is primarily concentrated on substantive problem-solving issues. Finally, we discuss a hypothesis about how analogy might serve as an uncertainty reduction strategy in naturalistic complex problem solving.

2011

On the benefits and pitfalls of analogies for innovative design: Ideation performance based on analogical distance, commonness, and modality of examples Chan, Joel, Fu, Katherine, Schunn, Christian. D., Cagan, Jonathan, Wood, Kristin L., and Kotovsky, Kenneth Journal of Mechanical Design 2011 [Abstract] [PDF]
Drawing inspiration from examples by analogy can be a powerful tool for innovative design during conceptual ideation but also carries the risk of negative design outcomes (e.g., design fixation), depending on key properties of examples. Understanding these properties is critical for effectively harnessing the power of analogy. The current research explores how variations in analogical distance, commonness, and representa- tion modality influence the effects of examples on conceptual ideation. Senior-level engi- neering students generated solution concepts for an engineering design problem with or without provided examples drawn from the U.S. Patent database. Examples were crossed by analogical distance (near-field vs. far-field), commonness (more vs. less-common), and modality (picture vs. text). A control group that received no examples was included for comparison. Effects were examined on a mixture of ideation process and product var- iables. Our results show positive effects of far-field and less-common examples on novelty and variability in quality of solution concepts. These effects are not modulated by modal- ity. However, detailed analyses of process variables suggest divergent inspiration path- ways for far-field vs. less-common examples. Additionally, the combination of far-field, less-common examples resulted in more novel concepts than in the control group. These findings suggest guidelines for the effective design and implementation of design-by-anal- ogy methods, particularly a focus on far-field, less-common examples during the ideation process.