With the advent of technology, we now have an unprecedented amount of information available online. This can be used by decision-makers, the civically-engaged, journalists, and researchers alike, to inform themselves. However, doing so requires navigating and parsing a complicated web of disparate resources (such as news reports, scientific articles, and social media), which can make it hard to ‘see the forest for the trees’. Although we have made great advances in how information can be disseminated, there is a lack of technological support to integrate, evaluate, contest, and reason about it (Rahwan, 2017). Argumentation online is certainly possible—and common, but typically only adds to the ever-growing torrent of information, leaving behind lengthy disorganized threads, interspersed with random Internet banter. Since anyone can contribute, verifying the validity of statements found online is extremely time-consuming—time most people do not have or are unwilling to give up leisure time for (Feldman et al., 2018). As a result, many sources act like ‘echo chambers’: discussions of opposing views are inconveniently segregated, and ill-founded ideas can propagate freely as they remain unopposed (Stroud, 2008; Mercier and Sperber, 2011). It should therefore not come as a surprise that terms like ‘misinformation’, ‘fake news’, and ‘filter bubbles’ permeate current discussions on the state of the Internet (Baker, 2017).
Inspired by argumentation theory (Van Eemeren et al., 1999; Walton, 2009), research on the border of logic, philosophy, and computer science has taken up the challenge to create better tools to disseminate, structure, and analyze rational thought, collectively called argument technologies. These target a wide variety of application areas, including conflict resolution, legal argumentation in law, discussing and documenting design rationale, and sensemaking (Scheuer et al., 2010; Schneider et al., 2013). In this paper, we focus on collaborative, publicly available, web-based technologies with explicit support for interlocutors to view and contribute to argumentative discourse. While several such technologies exist (Schneider et al., 2013), they remain a niche outside of research; support for argumentation on mainstream social networks is limited to basic functionality such as up or down voting. Some researchers argue that the primary obstacle to more widespread adoption is a mismatch between the informal nature of online discussions and the highly formal (structured) functionality current argument technologies provide, i.e., usability (Schneider et al., 2013; Paglieri, 2017).
In this paper, we introduce the design of a website for collaborative argumentative discussion (named Socratrees), exploring the delicate balance between structure—an integral part of argumentation—and usability. Prior argument technologies typically rely on complex ontologies to represent claims and relationships between them (e.g., issue, position, challenge, justification, agreement, etc.). In contrast, the user interface introduced in this paper reduces argumentation to but three core concepts for the user to understand: (1) statements, that can (2) support and (3) oppose one another. Our primary goal is to let users shape their own arguments by collaboratively aggregating all information relevant to a given statement in one location. As opposed to prior work (e.g., (Snaith et al., 2010; Schneider et al., 2007)), our focus is less on analyzing a single argument in great detail and more on providing the necessary structure to represent many (competing) arguments side by side, without forming judgment as to which one is sound. We evaluate how well our design supports users to collaboratively integrate, evaluate, contest, and reason about information online during a six-week-long exploratory study.
2. Argumentation theory
To understand argumentation theory, it is important to understand how it differs from formal logic. Formal deductive logic falls short when trying to evaluate the quality of arguments expressed in ordinary everyday language (such as political discourse) (Van Eemeren et al., 1999; Walton, 2009). Practitioners and teachers of logic started challenging the traditional ideals of validity and soundness and it was soon recognized that “[formal logic] had in mind one important subset of arguments, but the realm of argumentation was much broader” (Blair and Johnson, 1987a). Essentially, the logic of argumentation must be distinguished from formal logic which concerns itself solely with inference/implication; instead, argumentation must be seen as dialectical—a process with arguments as a product of which both sides need to be investigated (for and against) to see how they interact (Blair and Johnson, 1987a).
Today, argumentation theory is an umbrella term for studying the “production, analysis and evaluation of argumentation” by adopting both descriptive and normative methods, i.e., by evaluating argumentative discourse empirically (as it occurs) as well as reflecting on the necessary criteria for reasonable argumentation (Van Eemeren et al., 1999). Given our focus on providing technological support for argumentative discourse, we are particularly interested in a normative approach since it can prescribe the necessary functionality to enable more effective argumentation. Therefore, informal logic–defined as “the normative study of argument” (Blair and Johnson, 1987b)—became a logical choice as the main driving force behind our design.
An argument is a set of statements (propositions), made up of three parts, a conclusion, a set of premises, and an inference from the premises to the conclusion. An argument can be supported by other arguments, or it can be attacked by other arguments, and by raising critical questions about it.
Adequacy of premises and inferences in informal logic is less strict than in formal logic. Rather than validity and soundness, Blair and Johnson (Blair and Johnson, 1987a) argue for: (1) acceptability (start with premises the audience is willing to accept), (2) relevance (premises ought to be relevant to the conclusion), and (3) sufficiency (premises ought to provide sufficient support for the conclusion). Premises can either work together to form a linked argument or contribute to the conclusion independently, whereby they form a convergent argument. Argumentation schemes prescribe commonly used forms of linked arguments in which each premise plays a specific role, e.g., an argument ‘from expert opinion’ or ‘from analogy’.
With these ‘rules’ of informal logic in mind, there are several ways of attacking an argument (Walton, 2009). First, argumentation schemes have associated critical questions, e.g., of an argument from expert opinion you can ask, “Is the premise consistent with what other experts in the field assert?” Considering the acceptability criteria, you can either question one of the premises or form a counter-argument which concludes the opposite (refutation). It is also important to note that arguments can attack one another. Lastly, you can argue that a premise is not relevant to the given conclusion, or point out logical fallacies in reasoning.
Based on multiple studies in argumentation theory there is a commonly held belief that humans are inherently bad at reasoning and argumentation, highlighting the need to make ‘critical thinking’ and similar topics part of the core curriculum in education (Scheuer et al., 2010). However, more recent work sketches a less bleak picture of the ‘layman’ arguer (Paglieri, 2017; Mercier and Sperber, 2011). When people are sufficiently motivated, i.e., when argumentation occurs in a dialogical context as opposed to a decontextualized and abstract task, they can evaluate arguments quite accurately. Furthermore, in contrast to studies that show that people are bad at producing arguments (e.g., by succumbing to confirmation bias), they form good arguments when challenged and evidence is made available to them. In other words, people are better at evaluating others’ arguments than producing their own. Therefore, argumentation is most effective in groups with heterogeneous views (Mercier and Sperber, 2011):
… when reasoning is used in a more felicitous context—that is, in arguments among people who disagree but have a common interest in the truth—the confirmation bias contributes to an efficient form of division of cognitive labor.
3. Argument technologies
Part of the roots of argument technologies can be traced back to some of the earliest work in computer science (Shum, 2003):
It is of particular note that ‘founding fathers’ of today’s interactive computing such as Bush and Engelbart envisaged argument construction and analysis as a key objective for the intellectual technologies they were conceiving.
Engelbart’s NLS (Engelbart and English, 1968) (introducing many concepts of personal computing) was designed to ‘augment human intellect’ and Bush’s Memex (frequently cited as anticipating the Internet) was introduced ‘as we may think’ (Bush, 1945). However, given the extensive scope of technologies supporting reasoning that followed (Scheuer et al., 2010), we limit ourselves to reviewing those that are publicly available, web-based, collaborative, and include explicit support for argumentation. Schneider et al. (Schneider et al., 2013) have conveniently reviewed exactly such systems (37 in total) and furthermore include a broader discussion of argument technologies. We direct the reader to this excellent review for a complete description of related work.
Here, we provide an overview of the most relevant systems related to Socratrees, while omitting others which might overlap in functionality but target different use cases (e.g., ‘IBIS-like’ decision-support systems (Conklin and Begeman, 1988), knowledge maps, and public opinion polling tools like Opinion Space (Faridani et al., 2010)). Specifically, we focus on systems that target ‘information seeking’ dialogues in which the goal is to find or share arguments related to a common topic of interest (Walton, 2009).
Common functionality in such systems is to break up arguments into smaller pieces (going by various names, such as statements, premises, claims, and ideas) and specifying connections between them which describe their relationship (e.g., ‘supports’, ‘attacks’, or ‘is similar to’). Relations which denote inferences construct the argument. The resulting underlying data structure is a graph of statements which provides an argumentative overview of a specific discussion.
3.1. Representation and structure
Although arguments essentially form a graph, they can be visualized in a number of different ways; systems have been classified aslinear, threaded, container, graph, matrix, or a combination thereof (Scheuer et al., 2010). Matrix views are extremely uncommon and therefore will not be discussed here.
Linear and threaded representations are most in line with those used in traditional social media, e.g., blogs and sites supporting conversation threading such as Reddit (sys, 2005) respectively. Argument technologies extend on this. For example, Rich Trellis (Chklovski et al., 2005) allows a mixture of arbitrary free text with the ability to annotate relations, resulting in a linear but formalized overview of an argument. Rich Trellis was later extended (yet simplified) to Tree Trellis by reducing the possible types of connections to pro and con and by supporting threading. Videolyzer (Diakopoulos et al., 2009) supports threaded discussions by adding and responding to time-anchored annotations in online informational videos (e.g., mark part of the transcript as a claim, express agreement, or indicate quality).
Container- and graph-based representations are the two most popular approaches in argument technologies. They differ primarily in how connections between statements are visualized. Containers group statements with similar connections to a common ‘root’ statement in a demarcated area, whereas graphs show each connection separately (Figure 1).
An example of a container-based representation is Debatepedia (sys, 2007), a wiki-based site on which pro and con arguments are grouped together per question on a specific topic. Most recently, Kialo (sys, gust) (likely the most popular argument technology at the time of writing) was introduced to fight the “Internet Shouting Factory”. Kialo displays all claims in support of/attacking another claim on opposing sides, and frames them in context of an overarching conversation topic. Although discrete connections are common, some systems provide more granularity. ConsiderIt (Kriplean et al., 2012) (now a commercial site (sys, 2016)) aggregates comments on political issues by asking users to register their degree of support on a sliding scale and adding pros/cons to motivate their position. As a result, users can filter pros and cons by degree of support.
Container representations provide a clearer, more concise, overview than graphs, at the cost of reducing the types of connections which can be represented. Therefore, argument analysis tools (with their primary focus on formalizing arguments) typically rely on graph representations. For example, OVA (Online Visualization of Argument) (Snaith et al., 2010) loads text (or a web page) side-by-side with a canvas displaying an associated argument map (Figure 2). Arguments are mapped by highlighting statements and drawing connections in between them on the canvas. Similar systems are Argunet (Schneider et al., 2007) (a desktop tool that supports sharing argument maps online) and the open-source project Arguman (sys, 2014b), although neither of these support mapping arguments from text like OVA. Common functionality is to allow specifying connections in more detail. For example, OVA supports common argumentation schemes such as ‘practical reasoning’ and ‘expert opinion’ (relied upon in Figure 2). A distinction can thus be made between argument technologies that primarily focus on argument analysis (favoring graph views) and those that focus on information seeking (favoring container views).
Lastly, we identify an additional important attribute of argument representation: whether or not a view can be traversed recursively, i.e., whether a statement can be made the center of attention by selecting it, thereby hiding all statements that are not directly connected to the selected statement and showing new ones that are. In contrast to displaying a complete argument map (which can typically be panned and zoomed), only a subset of statements are shown at any given time. Such views are necessary when considering large discussions which may contain cyclical connections and statements that are reused in multiple arguments—a full graph would soon become unintelligible. Most prior work represents single arguments with limited reuse of statements, thereby making recursive views surprisingly uncommon. However, Debategraph (sys, 2008; Macintosh, 2009) is a graph-based system that supports traversing arguments recursively and Kialo (sys, gust) supports decomposing arguments into nested hierarchies of pros and cons. Arguments can be explored in great detail by ‘drilling down’ into underlying claims.
3.2. Social interaction
Most argument analysis tools provide limited support for social interaction, other than editing arguments collaboratively. Conversely, systems with a reduced focus on analysis typically incorporate additional support for collaboration. For example, Cohere (Shum et al., 2008), TruthMapping (sys, 2006), and REASON (Introne, 2009) target a more general audience and allow users to express agreement with statements. Similarly, Kialo lets users assess the impact of claims on a 5-point scale. Such feedback is incorporated to indicate the strength of statements. Similar to modern social media, argument technologies can trigger notifications when new content is added to arguments a user has contributed to, and content can be moderated (e.g., reported as spam). Kialo seems the most advanced argument technology in regards to supporting social interactions.
3.3. Infrastructure and integration
Several ontologies have been introduced to define the structure of arguments (Schneider et al., 2013), of which the most promising, still in active development, is the Argument Interchange Format (AIF) (Reed et al., 2017). AIF enables sharing of arguments across different online services for argumentation, collectively called the ‘Argument Web’ (Bex et al., 2013). For example, AIFdb (sys, 2012a; Lawrence et al., 2012) is a public database for arguments, to which the ‘smoking dialog’ represented in Figure 2 was uploaded. ArguBlogging (sys, 2014a; Bex et al., 2014) is a browser plugin through which agreement or disagreement can be expressed anywhere online by highlighting text and posting a response to your personal blog and AIFdb. Similar systems (unrelated to the Argument Web) are rbutr (sys, 2012b) and hypothes.is (sys, 2013), providing a comparable infrastructure to support ‘open annotation’ anywhere on the web. Ontologies are key to automating argument mining and evaluation (Reed et al., 2017), both of which are considered outside of scope in this paper.
4. Design principles
Before presenting Socratrees, we will introduce key design principles which have influenced our design and contrast them with prior work.
4.1. Transparency first—inspire critical thinking
Our goal is not to prescribe what is true or false (i.e., to become a fact finder), but to provide transparency to arguments and to inspire critical thinking. Rather than presenting single arguments in great detail, we aim to provide an easily digestible overview of all relevant information in relation to a specific statement. Competing arguments thus live side-by-side and it is up to users to interpret them and draw their own conclusions. In other words, our main goal is not supporting ‘argumentation’ per se, but providing a record of the collaborative thought process in order to aid individual human reasoning. Such overviews facilitate distributed sensemaking (Fisher et al., 2012), and group reasoning in which argumentation theory predicts truth to win out (Mercier and Sperber, 2011).
4.2. Help finding common ground
Argumentation theory describes different types of dialog with differing goals: e.g., in a persuasion dialog the goal is to convince the other party, and in a deliberation dialog the goal is to decide on the best available course of action (Walton, 2009). Argument analysis systems (as discussed in related work) primarily target persuasion dialogs. In contrast, Socratrees targets information-seeking and inquiry dialogs, in which the goal is to exchange information and find and verify evidence respectively. By sharing statements in relation to one another (as supporting or opposing) and allowing users to express agreement with each, an overview becomes available of how well-supported statements are, the different reasons for believing or not believing in them, and how popular these are. Strong arguments on either side of a discussion indicate ‘common ground’, whereas statements on which opinions are divided reveal true points of contention.
4.3. Conducive to large-scale discussions
Debaters on comprehensive scientific problems are … like lawyers who have to take a side. Each of them intends to strengthen his own arguments and to weaken the arguments of the aggressor—but no judge is in the chair. … Finally we find ourselves all together in the same ship and are co-operating even when we think we are fighting one another.
To better support the cooperative nature of argumentation, we explore interaction techniques for large groups of users to collaboratively contribute to discussions comprising numerous statements, without hindering one another, and while retaining a suitable overview for people arriving later to, or having missed part of, the discussion.
Based on our review of web-based argument technologies, we identify three core challenges to supporting large-scale discussions online: (1) dealing with digressions in order to ensure focused argumentation, (2) supporting users to explore arguments at their own pace—without being overwhelmed by expert accounts, and (3) enabling/encouraging statement reuse in order to eliminate redundant discussions and to capitalize on prior knowledge. TruthMapping (sys, 2006) has a built-in mechanism to deal with digressions, and Kialo supports focused exploration of arguments by breaking them up into concise claims. However, statement reuse in prior systems is rare or simply unsupported. Kialo (sys, gust) supports reusing claims through the use of ‘symlinks’, but in practice this feature is hardly used. We observe that for statements to be reused, it is essential that they are ‘free of context’, i.e., that they can be interpreted outside of the context in which they were introduced. Prior systems do not enforce this.
Prior work has mostly enforced argument structure, requiring knowledge of argument analysis, at the cost of usability (Paglieri, 2017; Schneider et al., 2013). We believe that given a suitable medium, anyone can contribute meaningfully to argumentation online (Paglieri, 2017; Mercier and Sperber, 2011). Users should not be expected to know argumentation theory in order to start using argument technologies. Similar to Cohere (Shum et al., 2008), TruthMapping (sys, 2006), and Kialo (sys, gust), we target a more general audience and our goal is to find a suitable balance between imposing structure and allowing for free form argumentation. Furthermore, we aim to be non-discriminatory; it should always be possible for minorities and repressed groups to share unpopular or controversial beliefs.
Socratrees (Figure 3) is a collaborative, web-based, argument technology, targeting a general (non-expert) audience. The main user interface supports outlining arguments into hierarchies of supporting and opposing statements and relies on a container representation which can be traversed recursively to navigate between statements. Kialo (sys, gust) is the only other system combining a container representation with recursive navigation. In addition, Socratrees introduces two novel features to structure arguments: (1) statements can be represented both in a normal and negated form, and (2) statement relations (one statement supporting/opposing another) are considered statements in their own right which further supporting/opposing statements can be added to.
First, we present the structure of Socratrees and our motivation behind it in more detail, including how navigation is supported. Second, we explain the chosen visual metaphor and how we intend to evolve it over time. Lastly, we present community features and how we believe they can be used to inspire and scale up argumentation.
5.1. Structure and navigation
We have chosen to only support two types of relations between statements—supporting and opposing—similar to earlier argument technologies targeting a layman audience. We find this to be a suitable balance between structure and ease of use. Additional semantics are implicit and we trust users to be able to infer more concrete relations based on context. For example: “Pineapple does not belong on pizza” is supported by “Italians don’t put pineapple on pizza” and opposed by “Hawaiian pizza is very popular in Australia”. An explicit analysis of these arguments is unnecessary to understand the two points made. We have opted to label relations as ‘supporting/opposing’, in contrast to the more common ‘pro/contra’ and ‘supporting/attacking’. Early feedback indicated this alternative wording inspired animosity and one-sided thinking, counter to our design principles. For the very same reason, supporting and opposing statements are depicted using the same color, in contrast to prior work which mostly relies on green and red respectively. Although differing colors make sense in light of the ‘redundancy gain’ principle in HCI, we find it more important to highlight what statements have in common rather than what sets them apart; all statements contribute to the discussion in a meaningful way.
This similarity is further emphasized by the following observation: when considering the negated form (inverse) of a statement, all supporting statements become opposing statements, and vice versa. For example: “Governments should defend freedom of choice of its citizens” (as highlighted in Figure 3.H) becomes a supporting statement when considering that “Governments should not ban smoking” instead of “Governments should ban smoking”. Socratrees exploits this fact by supporting both a normal and negated form for statements. By clicking the ‘inverse’ icon (Figure 3.A), a negated form of the statement is shown and the supporting and opposing statements are swapped. A default “(not)” is prepended to the normal form of statements, but users can specify custom (more suitable) text for the negated form. This has two major advantages which reinforce our underlying design principles: (1) better statement reuse since the normal and inverted form of statements are structurally identical, and (2) discussions on unpopular or controversial ideas (e.g., holocaust denial) are directly linked to the arguments against. This makes suppressing them nonsensical as it would eliminate the arguments in support of what the majority of society believes to be factual or ‘true’. To our knowledge, Socratrees is the first system supporting both a normal and negated form for statements.
The last mechanism available in Socratrees to structure arguments is related to the requirement of relevance. Arguing whether a statement is relevant is an essential, common, part of argumentation, supported in traditional argument analysis tools by applying argumentation schemes and ‘warrants’ to statement relations. Systems targeting a more general audience like Kialo (sys, gust), Consider.it (sys, 2016), and TruthMapping (sys, 2006) do not provide support for this, presumably to sidestep the additional complexity in favor of usability. We try to find middle ground by repurposing the basic building block of argumentation—the statement—to represent statement relations as well. Concretely, when adding a statement as a supporting/opposing statement to statement , the system introduces this relation as the statement “ supports/opposes ” with the inverse form “ does not support/oppose ” (Figure 4). The newly created statement thus acts as the starting point for a discussion on whether or not the given statement is relevant in relation to the other, reusing concepts the user is already familiar with. It can also be used to construct certain linked arguments (statements working together to reach a conclusion), though not all. Similar to how underlying statements can be accessed recursively, the ‘statement relation’ can be accessed by clicking the ‘branch’ which connects the statement to the argument ‘tree’. Further branches within the statement relation are hidden since reasoning about them becomes overly complicated and the need to discuss them has not arisen during early testing.
For these structural features to work as intended, we introduce three requirements statements need to comply with:
Be concise: Keeping statements concise keeps discussions focused, makes statements more likely to be reused, and makes interpreting their negations easier. When possible, conjuncts should be avoided by splitting statements into two. As argued in the design of Tree Trellis (Chklovski et al., 2005), this comes at the expense of expressiveness, but also improves the ability to elaborate on arguments without restructuring them, and ensures ancillary points are not mixed with more central ones. To enforce this, statements have a maximum of 120 characters; a limit which did not inhibit us from expressing our own arguments during early testing.
Aim to be free of context: It should be possible to interpret a statement outside of the context within which it was introduced. For example: “Wind turbines are more effective” makes sense in relation to a statement about solar panels, but takes on a different meaning when used in the context of nuclear energy. To ensure effective reuse of statements, statements should thus be able to stand on their own and eliminate indexical references (e.g., ‘I’, ‘also’, ‘this’, and ‘that’).
Not be phrased as questions: Questions do not contribute concrete information to the structure of arguments, and rather indicate specific information is lacking or missing. In essence, the answers to questions are what constitute valid statements on Socratrees. This last requirement is controversial due to the unmistaken value of critical questions in discussions. However, critical questions can still be posted as comments on statements (Figure 3.B).
Navigating arguments is supported by traversing statements recursively. At any given time, only a single statement with its supporting and opposing statements is displayed. Opening a different statement (or statement relation) is done by clicking on it. While this ensures focus and allows users to explore parts of an argument most relevant to them (in line with our design principles), this becomes disorienting when following many links. To see which statements you followed to reach your current position in a tree, a ‘current branch’ section is displayed in the sidebar (Figure 3.C). Previously visited statements can be revisited by clicking on them, without erasing the stored navigation path. E.g., Figure 3 shows the path followed to reach the previously visited statement “Nicotine is an addictive drug”. Only when diverging from the presented path, the stored navigation path is overridden. The sidebar also shows which other statements the current statement is used in (Figure 3.D). On smaller screens (e.g., mobile phones) the sidebar becomes a collapsible side panel.
New statements can be added after having searched for existing statements first (Figure 3.E). An intermediate screen shows search results and an option to proceed with adding the statement in case the desired one is not found. Adding supporting and opposing statements works in a similar way (Figure 3.F). Importantly, the search results page also allows users to view and select the normal or negated form of statements. Enforcing ‘search first’ further encourages statement reuse. Statements are anonymous in order not to discourage people from expressing unpopular beliefs.
5.2. Visual metaphor
After considering other visual metaphors (‘branching rivers’ and ‘neural networks’), we eventually chose to represent statements and their supporting and opposing arguments asleaves on a tree. This is a visually rich metaphor that everyone can relate to and evokes a sense of tranquility which might counteract the adverse nature of argumentation. When you agree with (or believe in) a statement you can ‘give it water’, represented as droplets (Figure 3.G). Although our current exploitation of this metaphor is limited, we chose to represent statements as leaves since the physical properties of a leaf correlate nicely with those we need to represent. Statement relevance (the statement relation) is presented by the stalk leading up to the leaf. The stalk might be ‘broken’ when a statement is deemed irrelevant as determined by underlying statements. Leaves in turn can have different colors depending on their ‘health’. Once we have explored the properties of what makes a good or bad statement in more detail (in an exploratory study) we intend to include such visualizations.
5.3. Community features
Since we first wanted to explore our alternative structure for argumentative discourse, the number of features for social interaction available in this first iteration of Socratrees are limited. However, we implemented basic communication, notification, statistics, and moderation features which provide a strong basis for future work.
We recognize that the potential of Socratrees lies not in replacing existing discussion platforms, but augmenting them. Therefore—by design—we provide limited support for unstructured argumentation; free form comments can be added to each statement (Figure 3.B) to add unstructured thoughts, but their primary purpose is to question, clarify, and add related resources. Lengthy discussions are discouraged111This is inspired by comments on Stack Exchange (https://stackexchange.com/) which are seen as transient. Valuable comments should be incorporated in the main content of the site (questions and answers).. We count on external websites, less restrictive in form and embodying richer communities, to link to content created on Socratrees and extend on discussions there. To enable this, each statement has a representative URL (e.g., “/statement/657/governments-should-ban-smoking”), and users can link to individual comments and statement relations (Figure 3.H). Furthermore, for our exploratory study we integrated with a Reddit discussion board. Clicking a Reddit icon linked to the creation of a new Reddit post with a matching title and a link back to the specific content on Socratrees (Figure 3.I).
The first few statements new users post are posted as drafts (Figure 3.J). Draft statements are under review and cannot be added to or used as related statements until they are approved. We implemented this feature so that moderators can provide feedback to new users in case their statements do not comply with site guidelines (in comments). Moderators can also turn problematic statements into drafts. For now, moderators are predetermined, but we intend to adopt a reputation-based distributed moderation model similar to Stack Exchange (Mamykina et al., 2011).
Users can add ‘droplets’ to leafs or stalks (statement relations) they believe in. The accuracy of statements can thus be judged independently of their relevance. A leaf icon next to droplets indicates the total number of underlying statements (Figure 3.K). To express belief in the negated form of a statement the inverse view needs to be loaded first. This forces users to view underlying statements prior to expressing their disbelief (the equivalent of ‘down voting’ in traditional social media). We hope this may inspire users to clarify their disagreement, which we deem more constructive than mere down voting. You can only express belief in one form of a statement at a time. An icon indicates whether you have expressed belief in the inverse form of the currently shown statement (Figure 3.L). The result is an overview of all the statements you have formed an opinion on.
Adding droplets also acts as a subscribe mechanism; users receive notifications about changes to any of the statements they have expressed belief or disbelief in. We feel expressing your opinion should go hand in hand with a willingness to participate in a discussion about it, and have therefore opted to combine these two features into one. A top bar shows an inbox for notifications, in addition to a link to the user’s account, the number of times people have expressed agreement with statements added by the user (regardless of form), and the number of approved statements posted by the user (Figure 3.M).
6. Exploratory study
We evaluated Socratrees during a six-week-long exploratory study222Preregistration for this study, explaining the purpose and method used, is available on AsPredicted: https://aspredicted.org/blind.php?x=7p26xn involving 32 users. Our main research question was: how to strike the right balance between introducing functionality for structured argumentation and usability in order to obtain more widespread adoption of argument technologies?
The Socratrees website was announced to friends and colleagues in person, on social media, and several public websites (e.g., subreddits dedicated to argumentation and critical thinking). 238 unique users visited the site, out of which 59 signed up to participate in private beta and were granted access. Only users with access could view statements and contribute to the site. 27 users did not visit the site after having received access, based on which we conclude that 32 users in total participated (to some degree) in private beta.
During private beta, 19 users added 374 statements and 371 statement relations (so 13 users only browsed the site). However, it must be noted that the majority of statements were added by the authors of this paper: 246 statements and 286 relations. Thus, the remaining users added 128 statements and 85 relations. Figure 5 shows an overview of all connections between statements, laid out so that individual arguments and connections between them can be recognized. 45 statements (12%) were used more than once (as supporting/opposing). Negated forms of statements contributed to statement reuse and lead to a particularly interesting use case: at times, both forms of a statement were added to opposing sides of an argument, thereby highlighting key points of disagreement (e.g., “Global warming is/is not man-made” supports/opposes “Climate change is man-made” in Figure 5.A). The average text of approved statements was 58 characters long (, , , ).
All 32 active users received an online survey at the end of the study, which was filled out by 14. Our survey data shows a wide variety of users: the median age was 28.5 (, ,
), 8 male and 6 female, with backgrounds such as data science, HR consulting, IT, and project management. On a 5-point Likert scale, nobody specified familiarity with argumentation theory higher than ‘Neutral’ (, meaning ‘Not familiar’). The median number of hours users reported having used the site was 2 (, , ). Five central survey items made claims about the potential of Socratrees and had to be rated on a 5-point Likert scale with the option to provide optional free form feedback: Socratrees (Q1) “can help understanding arguments”; (Q2) “can help finding common ground”; (Q3) “is non-discriminatory”; (Q4) “can host conducive, non-inflammatory, discussions”; and (Q5) “inspires critical thinking”. The distribution of responses (Figure 6) indicates most users believe Socratrees has the potential to achieve the goals we originally laid out in our design principles. Multiple other users provided feedback via private communication channels and on the Reddit discussion board, which we also include in our findings.
(Q1) 11 users elaborated that the division of supporting/opposing statements provides a “convenient” and “quick overview” which “makes it easier to understand arguments” and “figure out why two opinions differ.” However, one user noted that “the writer’s definition of a term [might be] inconsistent with the reader’s”, potentially leading to conflicts in understanding. (Q2) Most users indicated that “[b]y separating arguments into small and separate discussions, it becomes easier to see where you agree/disagree” and you “are more likely to read both sides”. They noted that you are more likely to agree on a single statement than on a whole argument; “people can agree on having different value-sets.” (Q3) A common concern was Socratrees requires a “special skill set only common in people with higher educations”, e.g., having a “sense of how important details can be in a [statement]”. And, while it “allows for expressing [anonymous] opinions, regardless of their popularity”, two users expressed concerns about “counterfactual” statements. Allowing widespread, uncontrolled, discriminating statements “would run counter to the goal of being non-discriminatory.” (Q4) Four users shared the belief that “[t]he structure of the platform leaves little room for personal attacks and puts the topic at hand at the core of any discussion.” Three others argued that discussions are inherently “emotional” and will always elicit “strong reactions”. Socratrees can “temper [this] but can’t eliminate it”. (Q5) Users overwhelmingly agreed that “[s]tructured argumentation encourages … users to understand the relationship between statements.” Socratrees “forces” users to “think before [they] write”, make the implicit explicit, and “to be very precise in the relation between [statements]”. However, you “have to spend a tremendous amount of energy to molding your statements to this structure and all its rules”.
Lastly, we asked users what would encourage them to contribute more to a site like Socratrees. Their primary request was a “larger community” with “topics that are relevant to current issues.” Users want to see that “[their] statements are making a difference” and to this end envisioned features such as sharing statements from external websites (supporting “evidence-based links”), enhanced reward mechanisms (gamification), and identifying like-minded individuals based on tags for “ideologies and beliefs”. The lack of such features left some users wanting for a “purpose” and “clear goal”.
The authors of this paper used Socratrees extensively to structure their own arguments and moderated all content added to the site, based on which they garnered a thorough understanding of limitations to the current design.
First, comments were underused and not fully understood by users. Even though they were designed as a staging ground for thoughts that could not yet be phrased as concrete statements, users still expressed a desire to add less-structured argumentation. Comments thus need more exposure and additional mechanisms need to be put in place to go from unstructured to fully-structured argumentation.
Second, adding statements to statement relations (in order to discuss relevance) is confusing and particularly error-prone (Figure 4). While we maintain this is an essential feature, it requires keeping track of three statements and two statement relations in parallel, which can be cognitively demanding. Furthermore, it is not always clear whether two statements should be added as one arguing for the relevance of the other, vice versa, or both. For example, “All questions seek knowledge” supports “There are no stupid questions”, which is relevant because “Seeking knowledge cannot be stupid”. But, an alternate phrasing would be “Seeking knowledge cannot be stupid”, which is relevant because “All questions seek knowledge”. We see no immediate harm in statements ‘reinforcing’ one another like this and adding both forms, but these findings indicate the need for further (usability) testing and trying out alternate visualizations.
Third, very similar statements require entirely different arguments. Yet, by adding arguments, some users were implicitly changing the specificity of the statement they were contributing to. For example, statements regarding second-hand smoke were being raised in a discussion on whether “Governments should ban smoking”. This is more relevant in a discussion on whether “Governments should ban smoking in public spaces”. Such related yet disparate statements need better support to keep discussions more focused.
Lastly, writing statements that are ‘free of context’ is unintuitive and hard. Many statements added by users were not living up to this guideline. Suchman (Suchman, 1987) argues that “the communicative significance of a linguistic expression is always dependent on the circumstances of its use”, making it impossible to remove all dependencies on external context. But, in practice it should be possible to elaborate on meaning to such a degree that the likelihood of misunderstandings arising due to differences in interpretation becomes minimal. This, however, requires excessive reiteration of context in each supporting and opposing statement, which becomes longer and longer as statements relate to more specific situations. To resolve this, we envision ‘context tags’ which can be added to statements to introduce implicit context (with an associated definition). When adding a related statement they are inherited by default, but can be removed by users in case they feel a statement can be made more generic.
Overall, we conclude that the structure imposed on argumentation by Socratrees accomplishes our goals to make arguments transparent, inspire critical thinking, help find common ground, and be conducive to discussion—at the cost of making it harder to contribute to arguments. However, the observation that structuring one’s thoughts in ‘network structures’ is challenging is commonplace (Shum, 2003):
… argument mapping initially feels like learning a new foreign language, and the temptation is to lapse back into more familiar languages (conversational patterns and modes of writing). The tools can be made user friendly, and the notations lightweight and informal, but the human element of the system must co-evolve as well.
While users should not have to know argumentation theory (none of our respondents did) in order to participate in argumentation, it would be overly optimistic to hope to eliminate the ‘hard work’ that goes into formulating an effective argument. The very act of identifying statements and relationships between them is what defines critical thinking. Therefore, we can only hope to inspire users to engage in and learn this alternate form of literacy by providing better support and making the work more rewarding and fun.
However, this does not mean we intend to replace traditional writing or believe that all arguments should be constructed as argument maps. Socratrees is designed to augment linear text. Our long-term goal is to be able to highlight statements anywhere on the web (relying on an ‘open annotation’ infrastructure similar to hypothes.is (sys, 2013) and rbutr (sys, 2012b)) and linking them to structured argumentation. We feel this could effectively replace traditional comment sections on news articles, blog posts, or even scientific papers, and provide better support for people to critically evaluate, contest, and reason about information online. At a glance, the popularity of statements and how well-supported or controversial they are could be assessed, and more advanced statistics might be able to indicate how objective or biased an article is. Looking even further ahead, making arguments more focused and open to falsification has the potential of scaling up governance and by extension the democratic process (Rahwan, 2017).
- sys (2005) 2005. Reddit. https://reddit.com.
- sys (2006) 2006. TruthMapping. http://www.truthmapping.com.
- sys (2007) 2007. Debatepedia. http://www.debatepedia.org.
- sys (2008) 2008. Debategraph. https://debategraph.org/.
- sys (2012a) 2012a. AIFdb. http://www.aifdb.org.
- sys (2012b) 2012b. rbutr. http://rbutr.com.
- sys (2013) 2013. hypothes.is. https://web.hypothes.is.
- sys (2014a) 2014a. ArguBlogging. http://argublogging.com.
- sys (2014b) 2014b. Arguman. http://arguman.org.
- sys (2016) 2016. Consider.it. https://consider.it.
- sys (gust) 2017, August. Kialo. https://www.kialo.com.
- Baker (2017) David Baker. 2017. The internet is broken. https://www.wired.co.uk/article/is-the-internet-broken-how-to-fix-it
- Bex et al. (2013) Floris Bex, John Lawrence, Mark Snaith, and Chris Reed. 2013. Implementing the argument web. Commun. ACM 56, 10 (2013), 66–73.
- Bex et al. (2014) Floris Bex, Mark Snaith, John Lawrence, and Chris Reed. 2014. Argublogging: An application for the argument web. Web Semantics: Science, Services and Agents on the World Wide Web 25 (2014), 9–15.
- Blair and Johnson (1987a) J Anthony Blair and Ralph H Johnson. 1987a. Argumentation as dialectical. Argumentation 1, 1 (1987), 41–56.
- Blair and Johnson (1987b) J Anthony Blair and Ralph H Johnson. 1987b. The current state of informal logic. Informal Logic 9, 2 (1987).
- Bush (1945) Vannevar Bush. 1945. As we may think. The atlantic monthly 176, 1 (1945), 101–108.
- Chklovski et al. (2005) Timothy Chklovski, Varun Ratnakar, and Yolanda Gil. 2005. User interfaces with semi-formal representations: a study of designing argumentation structures. In Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 130–136.
- Conklin and Begeman (1988) Jeff Conklin and Michael L Begeman. 1988. gIBIS: A hypertext tool for exploratory policy discussion. ACM Transactions on Information Systems (TOIS) 6, 4 (1988), 303–331.
- Diakopoulos et al. (2009) Nicholas Diakopoulos, Sergio Goldenberg, and Irfan Essa. 2009. Videolyzer: quality analysis of online informational video for bloggers and journalists. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 799–808.
- Engelbart and English (1968) Douglas C Engelbart and William K English. 1968. A research center for augmenting human intellect. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I. ACM, 395–410.
- Faridani et al. (2010) Siamak Faridani, Ephrat Bitton, Kimiko Ryokai, and Ken Goldberg. 2010. Opinion space: a scalable tool for browsing online comments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1175–1184.
- Feldman et al. (2018) Lauren Feldman, Magdalena Wojcieszak, Natalie Jomini Stroud, and Bruce Bimber. 2018. Explaining Media Choice: The Role of Issue-Specific Engagement in Predicting Interest-Based and Partisan Selectivity. Journal of Broadcasting & Electronic Media 62, 1 (2018), 109–130.
- Fisher et al. (2012) Kristie Fisher, Scott Counts, and Aniket Kittur. 2012. Distributed sensemaking: improving sensemaking by leveraging the efforts of previous users. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 247–256.
- Introne (2009) Joshua E Introne. 2009. Supporting group decisions by mediating deliberation to improve information pooling. In Proceedings of the ACM 2009 international conference on Supporting group work. ACM, 189–198.
- Kriplean et al. (2012) Travis Kriplean, Jonathan Morgan, Deen Freelon, Alan Borning, and Lance Bennett. 2012. Supporting reflective public thought with considerIt. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 265–274.
- Lawrence et al. (2012) John Lawrence, Floris Bex, Chris Reed, and Mark Snaith. 2012. AIFdb: Infrastructure for the Argument Web.. In COMMA. 515–516.
- Macintosh (2009) Ann Macintosh. 2009. Moving Toward Intelligent Policy Development? IEEE Intelligent Systems 24, 5 (2009), 79–82.
- Mamykina et al. (2011) Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Björn Hartmann. 2011. Design lessons from the fastest q&a site in the west. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2857–2866.
- Mercier and Sperber (2011) Hugo Mercier and Dan Sperber. 2011. Why do humans reason? Arguments for an argumentative theory. Behavioral and brain sciences 34, 2 (2011), 57–74.
- Neurath (1940) Otto Neurath. 1940. Universal jargon and terminology. In Proceedings of the Aristotelian Society, Vol. 41. JSTOR, 127–148.
- Paglieri (2017) Fabio Paglieri. 2017. A Plea for Ecological Argument Technologies. Philosophy & Technology 30, 2 (2017), 209–238.
- Rahwan (2017) Iyad Rahwan. 2017. Towards Scalable Governance: Sensemaking and Cooperation in the Age of Social Media. Philosophy & Technology 30, 2 (2017), 161–178.
- Reed et al. (2017) Chris Reed, Katarzyna Budzynska, Rory Duthie, Mathilde Janier, Barbara Konat, John Lawrence, Alison Pease, and Mark Snaith. 2017. The argument web: An online ecosystem of tools, systems and services for argumentation. Philosophy & Technology 30, 2 (2017), 137–160.
- Scheuer et al. (2010) Oliver Scheuer, Frank Loll, Niels Pinkwart, and Bruce M McLaren. 2010. Computer-supported argumentation: A review of the state of the art. International Journal of Computer-supported collaborative learning 5, 1 (2010), 43–102.
- Schneider et al. (2007) David C Schneider, Christian Voigt, and Gregor Betz. 2007. Argunet-a software tool for collaborative argumentation analysis and research. In 7th Workshop on Computational Models of Natural Argument (CMNA VII).
- Schneider et al. (2013) Jodi Schneider, Tudor Groza, and Alexandre Passant. 2013. A review of argumentation for the social semantic web. Semantic Web 4, 2 (2013), 159–218.
- Shum (2003) Simon Buckingham Shum. 2003. The roots of computer supported argument visualization. In Visualizing argumentation. Springer, 3–24.
- Shum et al. (2008) Simon Buckingham Shum et al. 2008. Cohere: Towards web 2.0 argumentation. COMMA 8 (2008), 97–108.
- Snaith et al. (2010) Mark Snaith, John Lawrence, and Chris Reed. 2010. Mixed initiative argument in public deliberation. Online Deliberation (2010), 2.
- Stroud (2008) Natalie Jomini Stroud. 2008. Media use and political predispositions: Revisiting the concept of selective exposure. Political Behavior 30, 3 (2008), 341–366.
- Suchman (1987) Lucy A Suchman. 1987. Plans and situated actions: The problem of human-machine communication. Cambridge university press.
- Van Eemeren et al. (1999) Frans H Van Eemeren, Rob Grootendorst, et al. 1999. Developments in argumentation theory. (1999).
Argumentation theory: A very short introduction.
Argumentation in artificial intelligence. Springer, 1–22.