Non-Linear Editor for Text-Based Screencast

by   Jungkook Park, et al.
University of Minnesota

Screencasts, where computer screen is broadcast to a large audience on the web, are becoming popular as an online educational tool. Among various types of screencast content, popular are the contents that involve text editing, including computer programming. There are emerging platforms that support such text-based screencasts by recording every character insertion/deletion from the creator and reconstructing its playback on the viewer's screen. However, these platforms lack rich support for creating and editing the screencast itself, mainly due to the difficulty of manipulating recorded text changes; the changes are tightly coupled in sequence, thus modifying arbitrary part of the sequence is not trivial. We present a non-linear editing tool for text-based screencasts. With the proposed selective history rewrite process, our editor allows users to substitute an arbitrary part of a text-based screencast while preserving overall consistency of the rest of the text-based screencast.


page 1

page 2

page 3


De-rendering Stylized Texts

Editing raster text is a promising but challenging task. We propose to a...

Recurrent Inference in Text Editing

In neural text editing, prevalent sequence-to-sequence based approaches ...

A Revisit of Shape Editing Techniques: from the Geometric to the Neural Viewpoint

3D shape editing is widely used in a range of applications such as movie...

KBSET -- Knowledge-Based Support for Scholarly Editing and Text Processing

KBSET supports a practical workflow for scholarly editing, based on usin...

Typed Image-based Programming with Structure Editing

Many beloved programming systems are image-based: self-contained worlds ...

DE-Net: Dynamic Text-guided Image Editing Adversarial Networks

Text-guided image editing models have shown remarkable results. However,...

1 Introduction

Instructional screencasts are increasingly widespread as an online educational tool for a variety of topics. The medium of most screencasts is a video where the creator records their screen using a screen recording software. However, since the video is mainly a graphical, view-only medium, it is difficult to provide even a basic interaction with the in-video content such as dragging and copying a text shown in the video. To overcome this problem, there are emerging online platforms that support text-based screencast for demonstrating terminal sessions [5] or programming tutorials [6]

. Instead of recording the screen as a video, these platforms capture the insertions/deletions in a character level, cursor/selection changes, or other relevant event data from the content creator’s text editing activities to construct a text editing history, then provides the screencast to the viewers by reconstructing the text editing history with the data collected. This allows viewers to interact with the text in any given moment in the screencast, which is not possible with videos. For example, when viewing a programming tutorial screencast, a viewer may pause the playback, edit and run the code to better understand the content.

Despite the advantage of text-based screencast, many of the currently available systems are lack of rich support for creating such content compared to other media such as video. In particular, little effort has been devoted to supporting non-linear editing of text-based screencast, which is a method to randomly access and selectively edit intermediate parts of a content without a need to sequentially view and edit the content, which is a natural way of editing in the production of a video or audio contents today.

Unfortunately, implementing non-linear editing system for text editing history is technically challenging. Unlike video where each frame’s data is independent of each other, each revision of a text editing history is dependent on all of its prior changes. For that reason, rewriting a part of a text editing history involves adjusting the offsets (numeric values indicating in which position the text is edited) of all subsequent changes. Moreover, rewriting a certain consecutive range of a text editing history can introduce ambiguity in reconstructing later versions of the text. While several studies previously introduced non-linear editing systems for document change history [2] or code change history [3, 1, 4], most are based on snapshots or line-based diff system, which cannot give users a fine-grained control over the text editing history.

We propose selective history rewrite process that enables substituting an arbitrary part of a text-based screencast while preserving overall consistency of text editing history. Then we present a web-based non-linear editor for text-based screencasts, and describe our user interface design for selecting history range from text editing history.

2 Selective History Rewriting Process

A core component of our non-linear editor is the selective history rewriting process that enables substituting a history range – a consecutive part of a text editing history – with a new history while preserving overall consistency. The process consists of two successive steps, validation step and substitution step, given a history range to be substituted and new text changes that will be placed in the history range.

In validation step, we evaluate whether rewriting a history range introduces any ambiguity on the subsequent part of the original history. To correctly validate the ambiguity, we calculate the “effective area” of the given history range and mark any text changes inside the effective area in the subsequent part of the history as ambiguous. When any ambiguous text changes is found, the rewriting process is aborted and user intervention is needed to change the selection. Figure 1 shows one example which rewriting is not possible due to the violation in effective area.

After the validation, we go through substitution step where we calculate the effect of newly substituted history on the text editing history. Because the fore part of the original history are never affected by the substitution, we only re-calculate the offsets of the text changes in the subsequent part of the history. After the recalculation, we produce a new resulting screencast by combining the three partial text editing history – (1) the fore part of the original history, (2) the newly substituted history, and (3) re-calculated history of the subsequent part.

Figure 1: Removing the text change “remove brown” between the first and the second history introduces an ambiguity, resulting two valid sentences.
Figure 2: Selecting the same history range using two different selection method: Timeline-based selection (top) and Text-selection-based selection (bottom).
Figure 3: History rewrite mode (top) and the rewrited history with the string “black crow” in place of “brown fox” (bottom).

3 Text Editor with Selective History Rewriting

We built a web-based interface to demonstrate non-linear editing of text-based screencast and its user interaction. Our interface consists of two main components: History Slider that visualizes the editing history (upper left in the interface), and the text area where user can view and edit the text. The History Slider shows a timeline of the editing history along the x-axis, and the offset of text changes in y-axis.

The foremost challenge in designing the interface for selective history rewriting was to find ways to allow users to easily select a history range they want to rewrite. We propose two interaction techniques for selecting a history range: Timeline-Based Selection (TL-Sel) and Text-Selection-Based Selection (TS-Sel). Using either of the methods results in a selection of a contiguous history range. The selected range is visualized in the History Slider as yellow highlighted region as shown in Figure 2.

When user selects a history range using either TS-Sel or TL-Sel, the validation step of selective history rewriting process immediately checks whether editing the selected history range introduces ambiguity. In case the selected history range is not rewritable, the editor disables the “Select” button to prevent user from starting rewriting the selected history. For TS-Sel, the editor indicates when a selection is invalid by coloring the text selection in red.

After selecting a valid history range, clicking the “Select” button enters the editor into edit mode. The text area now shows the earliest version of the text within the selected history range. In the text area, the user can only edit the regions that correspond to the effective area of the selected history range. In other words, the user can only edit the areas where the changes were made during the selected history range. The editor visualizes where the user can and cannot edit by showing gray background color behind the text that are not editable, as shown at the top of Figure 3. Clicking the “Done” button applies the substitution step of selective history rewriting process and the editor returns to normal editing mode with the substituted history.

4 Conclusion and Future Work

We introduced a web-based non-linear editor for text-based screencast, which allows users to substitute an arbitrary part of a text-based screencast while preserving overall consistency of the text editing history with the proposed selective history rewrite process.

In the current design, we simply prevent users from selecting a history range if rewriting the history range causes ambiguity. We believe it is crucial to devise with a way to resolve the ambiguity in order to make the editor practically usable for general use cases. One way would be to ask users to choose from the possible options. Another interesting idea is that some domain-specific heuristic algorithm or machine learning technique could be used to automatically identify which possible outcome is possibly the correct one so that we can eliminate the ambiguity.

Moreover, more investigation is needed towards better visualization and navigation of text editing history. In our future study, we plan to find ways to improve the visualization to provide additional contextual information in the timeline, such as segmenting the timeline and labeling the segment with relevant information.


  • [1] Bird, C., Rigby, P. C., Barr, E. T., Hamilton, D. J., German, D. M., and Devanbu, P. The promises and perils of mining git. In Mining Software Repositories, 2009. MSR’09. 6th IEEE International Working Conference on, IEEE (2009), 1–10.
  • [2] Edwards, W. K., and Mynatt, E. D. Timewarp: Techniques for autonomous collaboration. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, ACM (1997), 218–225.
  • [3] Ginosar, S., Pombo, D., Fernando, L., Agrawala, M., and Hartmann, B. Authoring multi-stage code examples with editable code histories. In Proceedings of the 26th annual ACM symposium on User interface software and technology, ACM (2013), 485–494.
  • [4] Hayashi, S., Omori, T., Zenmyo, T., Maruyama, K., and Saeki, M. Refactoring edit history of source code. In Software Maintenance (ICSM), 2012 28th IEEE International Conference on, IEEE (2012), 617–620.
  • [5] Kulik, M., and contributors. asciinema., 2012.
  • [6] Scrimba. Scrimba., 2017.