Lessons in Reproducibility: Insights from NLP Studies in Materials Science

by   Xiangyun Lei, et al.

Natural Language Processing (NLP), a cornerstone field within artificial intelligence, has been increasingly utilized in the field of materials science literature. Our study conducts a reproducibility analysis of two pioneering works within this domain: "Machine-learned and codified synthesis parameters of oxide materials" by Kim et al., and "Unsupervised word embeddings capture latent knowledge from materials science literature" by Tshitoyan et al. We aim to comprehend these studies from a reproducibility perspective, acknowledging their significant influence on the field of materials informatics, rather than critiquing them. Our study indicates that both papers offered thorough workflows, tidy and well-documented codebases, and clear guidance for model evaluation. This makes it easier to replicate their results successfully and partially reproduce their findings. In doing so, they set commendable standards for future materials science publications to aspire to. However, our analysis also highlights areas for improvement such as to provide access to training data where copyright restrictions permit, more transparency on model architecture and the training process, and specifications of software dependency versions. We also cross-compare the word embedding models between papers, and find that some key differences in reproducibility and cross-compatibility are attributable to design choices outside the bounds of the models themselves. In summary, our study appreciates the benchmark set by these seminal papers while advocating for further enhancements in research reproducibility practices in the field of NLP for materials science. This balance of understanding and continuous improvement will ultimately propel the intersecting domains of NLP and materials science literature into a future of exciting discoveries.


page 1

page 2

page 3

page 4


Analyzing Research Trends in Inorganic Materials Literature Using NLP

In the field of inorganic materials science, there is a growing demand t...

A Systematic Review of Reproducibility Research in Natural Language Processing

Against the background of what has been termed a reproducibility crisis ...

The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP

This paper introduces the Human Evaluation Datasheet, a template for rec...

NLP Reproducibility For All: Understanding Experiences of Beginners

As natural language processing (NLP) has recently seen an unprecedented ...

Looking Through Glass: Knowledge Discovery from Materials Science Literature using Natural Language Processing

Most of the knowledge in materials science literature is in the form of ...

Code Replicability in Computer Graphics

Being able to duplicate published research results is an important proce...

Efficient transfer learning for NLP with ELECTRA

Clark et al. [2020] claims that the ELECTRA approach is highly efficient...

Please sign up or login with your details

Forgot password? Click here to reset