Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming

04/12/2021
by   Adam Paszke, et al.
0

We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, "pointful" style with little performance penalty.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 23

07/06/2020

A Term-Rewriting Semantics for Imperative Style Programming

Term rewriting systems have a simple syntax and semantics and facilitate...
12/31/2019

Introduction to Rank-polymorphic Programming in Remora (Draft)

Remora is a higher-order, rank-polymorphic array-processing programming ...
10/10/2017

A Lambda Calculus for Transfinite Arrays: Unifying Arrays and Streams

Array programming languages allow for concise and generic formulations o...
12/29/2019

Performance optimization and modeling of fine-grained irregular communication in UPC

The UPC programming language offers parallelism via logically partitione...
05/31/2019

Reference Capabilities for Safe Parallel Array Programming

The array is a fundamental data structure that provides an efficient way...
07/30/2016

TRIOT: Faster tensor manipulation in C++11

[abridged] Context: Multidimensional arrays are used by many different a...
10/27/2020

Extended abstract: Type oriented programming for task based parallelism

Writing parallel codes is difficult and exhibits a fundamental trade-off...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.