-
Generic Go to Go: Dictionary-Passing, Monomorphisation, and Hybrid
Authors:
Stephen Ellis,
Shuofei Zhu,
Nobuko Yoshida,
Linhai Song
Abstract:
Go is a popular statically-typed industrial programming language. To aid the type safe reuse of code, the recent Go release (Go 1.18) published on 15th March 2022 includes bounded parametric polymorphism via generic types. Go 1.18 implements generic types using combination of monomorphisation and call-graph based dictionary-passing called hybrid. This hybrid approach can be viewed as an optimised…
▽ More
Go is a popular statically-typed industrial programming language. To aid the type safe reuse of code, the recent Go release (Go 1.18) published on 15th March 2022 includes bounded parametric polymorphism via generic types. Go 1.18 implements generic types using combination of monomorphisation and call-graph based dictionary-passing called hybrid. This hybrid approach can be viewed as an optimised form of monomorphisation that statically generates specialised methods and types based on possible instantiations. A monolithic dictionary supplements information lost during monomorphisation, and it is structured according to the program's call graph. Unfortunately, the hybrid approach still suffers from code bloat, poor compilation speed, and limited code coverage.
In this paper we propose and formalise a new non-specialising call-site based dictionary-passing translation. Our call-site based translation creates individual dictionaries for each type parameter, with dictionary construction occurring in place of instantiation, overcoming the limitations of hybrid. We prove it correct using a novel and general bisimulation up to technique. To better understand how different generics translations approaches work in practice, we benchmark five translators, Go 1.18, two existing monomorphisation translators, our dictionary-passing translator, and erasure translator. Our findings reveal several suggestions for improvements for Go 1.18 -- specifically how to overcome the expressiveness limitations of generic Go, and improve compile time and compiled code size performance of Go 1.18.
△ Less
Submitted 10 October, 2022; v1 submitted 14 August, 2022;
originally announced August 2022.
-
Is MC Dropout Bayesian?
Authors:
Loic Le Folgoc,
Vasileios Baltatzis,
Sujal Desai,
Anand Devaraj,
Sam Ellis,
Octavio E. Martinez Manzanera,
Arjun Nair,
Huaqi Qiu,
Julia Schnabel,
Ben Glocker
Abstract:
MC Dropout is a mainstream "free lunch" method in medical imaging for approximate Bayesian computations (ABC). Its appeal is to solve out-of-the-box the daunting task of ABC and uncertainty quantification in Neural Networks (NNs); to fall within the variational inference (VI) framework; and to propose a highly multimodal, faithful predictive posterior. We question the properties of MC Dropout for…
▽ More
MC Dropout is a mainstream "free lunch" method in medical imaging for approximate Bayesian computations (ABC). Its appeal is to solve out-of-the-box the daunting task of ABC and uncertainty quantification in Neural Networks (NNs); to fall within the variational inference (VI) framework; and to propose a highly multimodal, faithful predictive posterior. We question the properties of MC Dropout for approximate inference, as in fact MC Dropout changes the Bayesian model; its predictive posterior assigns $0$ probability to the true model on closed-form benchmarks; the multimodality of its predictive posterior is not a property of the true predictive posterior but a design artefact. To address the need for VI on arbitrary models, we share a generic VI engine within the pytorch framework. The code includes a carefully designed implementation of structured (diagonal plus low-rank) multivariate normal variational families, and mixtures thereof. It is intended as a go-to no-free-lunch approach, addressing shortcomings of mean-field VI with an adjustable trade-off between expressivity and computational complexity.
△ Less
Submitted 8 October, 2021;
originally announced October 2021.
-
The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification
Authors:
Vasileios Baltatzis,
Kyriaki-Margarita Bintsi,
Loic Le Folgoc,
Octavio E. Martinez Manzanera,
Sam Ellis,
Arjun Nair,
Sujal Desai,
Ben Glocker,
Julia A. Schnabel
Abstract:
Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the imp…
▽ More
Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the impact of individual contributions. When analyzing seven recent works, however, we find that each employs a different data selection process, leading to largely varying total number of samples and ratios between benign and malignant cases. As each subset will have different characteristics with varying difficulty for classification, a direct comparison between the proposed methods is thus not always possible, nor fair. We study the particular effect of truthing when aggregating labels from multiple experts. We show that specific choices can have severe impact on the data distribution where it may be possible to achieve superior performance on one sample distribution but not on another. While we show that we can further improve on the state-of-the-art on one sample selection, we also find that on a more challenging sample selection, on the same database, the more advanced models underperform with respect to very simple baseline methods, highlighting that the selected data distribution may play an even more important role than the model architecture. This raises concerns about the validity of claimed methodological contributions. We believe the community should be aware of these pitfalls and make recommendations on how these can be avoided in future work.
△ Less
Submitted 11 August, 2021;
originally announced August 2021.
-
The Effect of the Loss on Generalization: Empirical Study on Synthetic Lung Nodule Data
Authors:
Vasileios Baltatzis,
Loic Le Folgoc,
Sam Ellis,
Octavio E. Martinez Manzanera,
Kyriaki-Margarita Bintsi,
Arjun Nair,
Sujal Desai,
Ben Glocker,
Julia A. Schnabel
Abstract:
Convolutional Neural Networks (CNNs) are widely used for image classification in a variety of fields, including medical imaging. While most studies deploy cross-entropy as the loss function in such tasks, a growing number of approaches have turned to a family of contrastive learning-based losses. Even though performance metrics such as accuracy, sensitivity and specificity are regularly used for t…
▽ More
Convolutional Neural Networks (CNNs) are widely used for image classification in a variety of fields, including medical imaging. While most studies deploy cross-entropy as the loss function in such tasks, a growing number of approaches have turned to a family of contrastive learning-based losses. Even though performance metrics such as accuracy, sensitivity and specificity are regularly used for the evaluation of CNN classifiers, the features that these classifiers actually learn are rarely identified and their effect on the classification performance on out-of-distribution test samples is insufficiently explored. In this paper, motivated by the real-world task of lung nodule classification, we investigate the features that a CNN learns when trained and tested on different distributions of a synthetic dataset with controlled modes of variation. We show that different loss functions lead to different features being learned and consequently affect the generalization ability of the classifier on unseen data. This study provides some important insights into the design of deep learning solutions for medical imaging tasks.
△ Less
Submitted 10 August, 2021;
originally announced August 2021.
-
Bayesian analysis of the prevalence bias: learning and predicting from imbalanced data
Authors:
Loic Le Folgoc,
Vasileios Baltatzis,
Amir Alansary,
Sujal Desai,
Anand Devaraj,
Sam Ellis,
Octavio E. Martinez Manzanera,
Fahdi Kanavati,
Arjun Nair,
Julia Schnabel,
Ben Glocker
Abstract:
Datasets are rarely a realistic approximation of the target population. Say, prevalence is misrepresented, image quality is above clinical standards, etc. This mismatch is known as sampling bias. Sampling biases are a major hindrance for machine learning models. They cause significant gaps between model performance in the lab and in the real world. Our work is a solution to prevalence bias. Preval…
▽ More
Datasets are rarely a realistic approximation of the target population. Say, prevalence is misrepresented, image quality is above clinical standards, etc. This mismatch is known as sampling bias. Sampling biases are a major hindrance for machine learning models. They cause significant gaps between model performance in the lab and in the real world. Our work is a solution to prevalence bias. Prevalence bias is the discrepancy between the prevalence of a pathology and its sampling rate in the training dataset, introduced upon collecting data or due to the practioner rebalancing the training batches. This paper lays the theoretical and computational framework for training models, and for prediction, in the presence of prevalence bias. Concretely a bias-corrected loss function, as well as bias-corrected predictive rules, are derived under the principles of Bayesian risk minimization. The loss exhibits a direct connection to the information gain. It offers a principled alternative to heuristic training losses and complements test-time procedures based on selecting an operating point from summary curves. It integrates seamlessly in the current paradigm of (deep) learning using stochastic backpropagation and naturally with Bayesian models.
△ Less
Submitted 31 July, 2021;
originally announced August 2021.
-
Bayesian Sampling Bias Correction: Training with the Right Loss Function
Authors:
L. Le Folgoc,
V. Baltatzis,
A. Alansary,
S. Desai,
A. Devaraj,
S. Ellis,
O. E. Martinez Manzanera,
F. Kanavati,
A. Nair,
J. Schnabel,
B. Glocker
Abstract:
We derive a family of loss functions to train models in the presence of sampling bias. Examples are when the prevalence of a pathology differs from its sampling rate in the training dataset, or when a machine learning practioner rebalances their training dataset. Sampling bias causes large discrepancies between model performance in the lab and in more realistic settings. It is omnipresent in medic…
▽ More
We derive a family of loss functions to train models in the presence of sampling bias. Examples are when the prevalence of a pathology differs from its sampling rate in the training dataset, or when a machine learning practioner rebalances their training dataset. Sampling bias causes large discrepancies between model performance in the lab and in more realistic settings. It is omnipresent in medical imaging applications, yet is often overlooked at training time or addressed on an ad-hoc basis. Our approach is based on Bayesian risk minimization. For arbitrary likelihood models we derive the associated bias corrected loss for training, exhibiting a direct connection to information gain. The approach integrates seamlessly in the current paradigm of (deep) learning using stochastic backpropagation and naturally with Bayesian models. We illustrate the methodology on case studies of lung nodule malignancy grading.
△ Less
Submitted 24 June, 2020;
originally announced June 2020.
-
Animal social networks: an introduction for complex systems scientists
Authors:
Josefine Bohr Brask,
Samuel Ellis,
Darren P Croft
Abstract:
Many animals live in societies where individuals frequently interact socially with each other. The social structures of these systems can be studied in depth by means of network analysis. A large number of studies on animal social networks in many species have in recent years been carried out in the biological research field of animal behaviour and have provided new insights into behaviour, ecolog…
▽ More
Many animals live in societies where individuals frequently interact socially with each other. The social structures of these systems can be studied in depth by means of network analysis. A large number of studies on animal social networks in many species have in recent years been carried out in the biological research field of animal behaviour and have provided new insights into behaviour, ecology, and social evolution. This line of research is currently not so well connected to the field of complex systems as could be expected. The purpose of this paper is to provide an introduction to animal social networks for complex systems scientists and highlight areas of synergy. We believe that an increased integration of animal social networks with the interdisciplinary field of complex systems and networks would be beneficial for various reasons. Increased collaboration between researchers in this field and biologists studying animal social systems could be valuable in solving challenges that are of importance to animal social network research. Furthermore, animal social networks provide the opportunity to investigate hypotheses about complex systems across a range of natural real-world social systems. In this paper, we describe what animal social networks are and main research themes where they are studied; we give an overview of the methods commonly used to study animal social networks; we highlight challenges in the study of animal social networks where complex systems expertise may be particularly valuable; and we consider aspects of animal social networks that may be of particular interest to complex systems researchers. We hope that this will help to facilitate further interdisciplinary collaborations involving animal social networks, and further integration of these networks into the field of complex systems.
△ Less
Submitted 19 February, 2021; v1 submitted 19 May, 2020;
originally announced May 2020.
-
MolSSI and BioExcel Workflow Workshop 2018 Report
Authors:
Levi N. Naden,
Sam Ellis,
Shantenu Jha
Abstract:
Workflows in biomolecular science are very important as they are intricately intertwined with the scientific outcomes, as well as algorithmic and methodological innovations. The use and effectiveness of workflow tools to meet the needs of the biomolecular science community is varied. MolSSI co-organized a biomolecular workflows workshop in December 2018 with the goal of identifying specific softwa…
▽ More
Workflows in biomolecular science are very important as they are intricately intertwined with the scientific outcomes, as well as algorithmic and methodological innovations. The use and effectiveness of workflow tools to meet the needs of the biomolecular science community is varied. MolSSI co-organized a biomolecular workflows workshop in December 2018 with the goal of identifying specific software gaps and opportunities for improved workflow practices. This report captures presentations and discussion from that workshop. The workshop participants were primary tools developers, along with "neutral observers" and some biomolecular domain scientists. After contextualizing and motivating the workshop, the report covers the existing roles and emerging trends in how workflow systems are utilized. A few recurring observations are presented as recommendations for improving the use and effectiveness of workflow tools. The tools presented are discussed in Appendix B.
△ Less
Submitted 28 May, 2019;
originally announced May 2019.
-
Robust Chemical Circuits
Authors:
Samuel J. Ellis,
Titus H. Klinge,
James I. Lathrop
Abstract:
We introduce a new motif for constructing robust digital logic circuits using input/output chemical reaction networks. These chemical circuits robustly handle perturbations in input signals, initial concentrations, rate constants, and measurements. In particular, we show that all combinatorial circuits and several sequential circuits enjoy this robustness. Our results complement existing literatur…
▽ More
We introduce a new motif for constructing robust digital logic circuits using input/output chemical reaction networks. These chemical circuits robustly handle perturbations in input signals, initial concentrations, rate constants, and measurements. In particular, we show that all combinatorial circuits and several sequential circuits enjoy this robustness. Our results complement existing literature in the following three ways: (1) our logic gates read their inputs catalytically which make `fanout' gates unnecessary; (2) formal requirements and rigorous proofs of satisfaction are provided for each circuit; and (3) robustness of every circuit is closed under modular composition.
△ Less
Submitted 21 August, 2018;
originally announced August 2018.
-
The Effect of Focal Distance, Age, and Brightness on Near-Field Augmented Reality Depth Matching
Authors:
Gujot Singh,
Stephen R. Ellis,
J. Edward Swan II
Abstract:
Many augmented reality (AR) applications operate within near-field reaching distances, and require matching the depth of a virtual object with a real object. The accuracy of this matching was measured in three experiments, which examined the effect of focal distance, age, and brightness, within distances of 33.3 to 50 cm, using a custom-built AR haploscope. Experiment I examined the effect of foca…
▽ More
Many augmented reality (AR) applications operate within near-field reaching distances, and require matching the depth of a virtual object with a real object. The accuracy of this matching was measured in three experiments, which examined the effect of focal distance, age, and brightness, within distances of 33.3 to 50 cm, using a custom-built AR haploscope. Experiment I examined the effect of focal demand, at the levels of collimated (infinite focal distance), consistent with other depth cues, and at the midpoint of reaching distance. Observers were too young to exhibit age-related reductions in accommodative ability. The depth matches of collimated targets were increasingly overestimated with increasing distance, consistent targets were slightly underestimated, and midpoint targets were accurately estimated. Experiment II replicated Experiment I, with older observers. Results were similar to Experiment I. Experiment III replicated Experiment I with dimmer targets, using young observers. Results were again consistent with Experiment I, except that both consistent and midpoint targets were accurately estimated. In all cases, collimated results were explained by a model, where the collimation biases the eyes' vergence angle outwards by a constant amount. Focal demand and brightness affect near-field AR depth matching, while age-related reductions in accommodative ability have no effect.
△ Less
Submitted 30 November, 2017;
originally announced December 2017.
-
Runtime Fault Detection in Programmed Molecular Systems
Authors:
Samuel J. Ellis,
Titus H. Klinge,
James I. Lathrop,
Jack H. Lutz,
Robyn R. Lutz,
Andrew S. Miner,
Hugh D. Potter
Abstract:
Watchdog timers are devices that are commonly used to monitor the health of safety-critical hardware and software systems. Their primary function is to raise an alarm if the monitored systems fail to emit periodic "heartbeats" that signal their well-being. In this paper we design and verify a molecular watchdog timer for monitoring the health of programmed molecular nanosystems. This raises new ch…
▽ More
Watchdog timers are devices that are commonly used to monitor the health of safety-critical hardware and software systems. Their primary function is to raise an alarm if the monitored systems fail to emit periodic "heartbeats" that signal their well-being. In this paper we design and verify a molecular watchdog timer for monitoring the health of programmed molecular nanosystems. This raises new challenges because our molecular watchdog timer and the system that it monitors both operate in the probabilistic environment of chemical kinetics, where many failures are certain to occur and it is especially hard to detect the absence of a signal.
Our molecular watchdog timer is the result of an incremental design process that uses goal-oriented requirements engineering, simulation, stochastic analysis, and software verification tools. We demonstrate the molecular watchdog's functionality by having it monitor a molecular oscillator. Both the molecular watchdog timer and the oscillator are implemented as chemical reaction networks, which are the current programming language of choice for many molecular programming applications.
△ Less
Submitted 23 July, 2018; v1 submitted 25 October, 2017;
originally announced October 2017.