Showing 1–11 of 11 results for author: Ellis, S

Generic Go to Go: Dictionary-Passing, Monomorphisation, and Hybrid

Authors: Stephen Ellis, Shuofei Zhu, Nobuko Yoshida, Linhai Song

Abstract: Go is a popular statically-typed industrial programming language. To aid the type safe reuse of code, the recent Go release (Go 1.18) published on 15th March 2022 includes bounded parametric polymorphism via generic types. Go 1.18 implements generic types using combination of monomorphisation and call-graph based dictionary-passing called hybrid. This hybrid approach can be viewed as an optimised… ▽ More Go is a popular statically-typed industrial programming language. To aid the type safe reuse of code, the recent Go release (Go 1.18) published on 15th March 2022 includes bounded parametric polymorphism via generic types. Go 1.18 implements generic types using combination of monomorphisation and call-graph based dictionary-passing called hybrid. This hybrid approach can be viewed as an optimised form of monomorphisation that statically generates specialised methods and types based on possible instantiations. A monolithic dictionary supplements information lost during monomorphisation, and it is structured according to the program's call graph. Unfortunately, the hybrid approach still suffers from code bloat, poor compilation speed, and limited code coverage. In this paper we propose and formalise a new non-specialising call-site based dictionary-passing translation. Our call-site based translation creates individual dictionaries for each type parameter, with dictionary construction occurring in place of instantiation, overcoming the limitations of hybrid. We prove it correct using a novel and general bisimulation up to technique. To better understand how different generics translations approaches work in practice, we benchmark five translators, Go 1.18, two existing monomorphisation translators, our dictionary-passing translator, and erasure translator. Our findings reveal several suggestions for improvements for Go 1.18 -- specifically how to overcome the expressiveness limitations of generic Go, and improve compile time and compiled code size performance of Go 1.18. △ Less

Submitted 10 October, 2022; v1 submitted 14 August, 2022; originally announced August 2022.

Comments: Full version of paper submitted to OOPSLA '22

Is MC Dropout Bayesian?

Authors: Loic Le Folgoc, Vasileios Baltatzis, Sujal Desai, Anand Devaraj, Sam Ellis, Octavio E. Martinez Manzanera, Arjun Nair, Huaqi Qiu, Julia Schnabel, Ben Glocker

Abstract: MC Dropout is a mainstream "free lunch" method in medical imaging for approximate Bayesian computations (ABC). Its appeal is to solve out-of-the-box the daunting task of ABC and uncertainty quantification in Neural Networks (NNs); to fall within the variational inference (VI) framework; and to propose a highly multimodal, faithful predictive posterior. We question the properties of MC Dropout for… ▽ More MC Dropout is a mainstream "free lunch" method in medical imaging for approximate Bayesian computations (ABC). Its appeal is to solve out-of-the-box the daunting task of ABC and uncertainty quantification in Neural Networks (NNs); to fall within the variational inference (VI) framework; and to propose a highly multimodal, faithful predictive posterior. We question the properties of MC Dropout for approximate inference, as in fact MC Dropout changes the Bayesian model; its predictive posterior assigns $0$ probability to the true model on closed-form benchmarks; the multimodality of its predictive posterior is not a property of the true predictive posterior but a design artefact. To address the need for VI on arbitrary models, we share a generic VI engine within the pytorch framework. The code includes a carefully designed implementation of structured (diagonal plus low-rank) multivariate normal variational families, and mixtures thereof. It is intended as a go-to no-free-lunch approach, addressing shortcomings of mean-field VI with an adjustable trade-off between expressivity and computational complexity. △ Less

Submitted 8 October, 2021; originally announced October 2021.

The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification

Authors: Vasileios Baltatzis, Kyriaki-Margarita Bintsi, Loic Le Folgoc, Octavio E. Martinez Manzanera, Sam Ellis, Arjun Nair, Sujal Desai, Ben Glocker, Julia A. Schnabel

Abstract: Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the imp… ▽ More Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the impact of individual contributions. When analyzing seven recent works, however, we find that each employs a different data selection process, leading to largely varying total number of samples and ratios between benign and malignant cases. As each subset will have different characteristics with varying difficulty for classification, a direct comparison between the proposed methods is thus not always possible, nor fair. We study the particular effect of truthing when aggregating labels from multiple experts. We show that specific choices can have severe impact on the data distribution where it may be possible to achieve superior performance on one sample distribution but not on another. While we show that we can further improve on the state-of-the-art on one sample selection, we also find that on a more challenging sample selection, on the same database, the more advanced models underperform with respect to very simple baseline methods, highlighting that the selected data distribution may play an even more important role than the model architecture. This raises concerns about the validity of claimed methodological contributions. We believe the community should be aware of these pitfalls and make recommendations on how these can be avoided in future work. △ Less

Submitted 11 August, 2021; originally announced August 2021.

Comments: Accepted at PRIME, MICCAI 2021

The Effect of the Loss on Generalization: Empirical Study on Synthetic Lung Nodule Data

Authors: Vasileios Baltatzis, Loic Le Folgoc, Sam Ellis, Octavio E. Martinez Manzanera, Kyriaki-Margarita Bintsi, Arjun Nair, Sujal Desai, Ben Glocker, Julia A. Schnabel

Abstract: Convolutional Neural Networks (CNNs) are widely used for image classification in a variety of fields, including medical imaging. While most studies deploy cross-entropy as the loss function in such tasks, a growing number of approaches have turned to a family of contrastive learning-based losses. Even though performance metrics such as accuracy, sensitivity and specificity are regularly used for t… ▽ More Convolutional Neural Networks (CNNs) are widely used for image classification in a variety of fields, including medical imaging. While most studies deploy cross-entropy as the loss function in such tasks, a growing number of approaches have turned to a family of contrastive learning-based losses. Even though performance metrics such as accuracy, sensitivity and specificity are regularly used for the evaluation of CNN classifiers, the features that these classifiers actually learn are rarely identified and their effect on the classification performance on out-of-distribution test samples is insufficiently explored. In this paper, motivated by the real-world task of lung nodule classification, we investigate the features that a CNN learns when trained and tested on different distributions of a synthetic dataset with controlled modes of variation. We show that different loss functions lead to different features being learned and consequently affect the generalization ability of the classifier on unseen data. This study provides some important insights into the design of deep learning solutions for medical imaging tasks. △ Less

Submitted 10 August, 2021; originally announced August 2021.

Comments: Accepted at iMIMIC, MICCAI 2021

Bayesian analysis of the prevalence bias: learning and predicting from imbalanced data

Authors: Loic Le Folgoc, Vasileios Baltatzis, Amir Alansary, Sujal Desai, Anand Devaraj, Sam Ellis, Octavio E. Martinez Manzanera, Fahdi Kanavati, Arjun Nair, Julia Schnabel, Ben Glocker

Abstract: Datasets are rarely a realistic approximation of the target population. Say, prevalence is misrepresented, image quality is above clinical standards, etc. This mismatch is known as sampling bias. Sampling biases are a major hindrance for machine learning models. They cause significant gaps between model performance in the lab and in the real world. Our work is a solution to prevalence bias. Preval… ▽ More Datasets are rarely a realistic approximation of the target population. Say, prevalence is misrepresented, image quality is above clinical standards, etc. This mismatch is known as sampling bias. Sampling biases are a major hindrance for machine learning models. They cause significant gaps between model performance in the lab and in the real world. Our work is a solution to prevalence bias. Prevalence bias is the discrepancy between the prevalence of a pathology and its sampling rate in the training dataset, introduced upon collecting data or due to the practioner rebalancing the training batches. This paper lays the theoretical and computational framework for training models, and for prediction, in the presence of prevalence bias. Concretely a bias-corrected loss function, as well as bias-corrected predictive rules, are derived under the principles of Bayesian risk minimization. The loss exhibits a direct connection to the information gain. It offers a principled alternative to heuristic training losses and complements test-time procedures based on selecting an operating point from summary curves. It integrates seamlessly in the current paradigm of (deep) learning using stochastic backpropagation and naturally with Bayesian models. △ Less

Submitted 31 July, 2021; originally announced August 2021.

Bayesian Sampling Bias Correction: Training with the Right Loss Function

Authors: L. Le Folgoc, V. Baltatzis, A. Alansary, S. Desai, A. Devaraj, S. Ellis, O. E. Martinez Manzanera, F. Kanavati, A. Nair, J. Schnabel, B. Glocker

Abstract: We derive a family of loss functions to train models in the presence of sampling bias. Examples are when the prevalence of a pathology differs from its sampling rate in the training dataset, or when a machine learning practioner rebalances their training dataset. Sampling bias causes large discrepancies between model performance in the lab and in more realistic settings. It is omnipresent in medic… ▽ More We derive a family of loss functions to train models in the presence of sampling bias. Examples are when the prevalence of a pathology differs from its sampling rate in the training dataset, or when a machine learning practioner rebalances their training dataset. Sampling bias causes large discrepancies between model performance in the lab and in more realistic settings. It is omnipresent in medical imaging applications, yet is often overlooked at training time or addressed on an ad-hoc basis. Our approach is based on Bayesian risk minimization. For arbitrary likelihood models we derive the associated bias corrected loss for training, exhibiting a direct connection to information gain. The approach integrates seamlessly in the current paradigm of (deep) learning using stochastic backpropagation and naturally with Bayesian models. We illustrate the methodology on case studies of lung nodule malignancy grading. △ Less

Submitted 24 June, 2020; originally announced June 2020.

Animal social networks: an introduction for complex systems scientists

Authors: Josefine Bohr Brask, Samuel Ellis, Darren P Croft

Abstract: Many animals live in societies where individuals frequently interact socially with each other. The social structures of these systems can be studied in depth by means of network analysis. A large number of studies on animal social networks in many species have in recent years been carried out in the biological research field of animal behaviour and have provided new insights into behaviour, ecolog… ▽ More Many animals live in societies where individuals frequently interact socially with each other. The social structures of these systems can be studied in depth by means of network analysis. A large number of studies on animal social networks in many species have in recent years been carried out in the biological research field of animal behaviour and have provided new insights into behaviour, ecology, and social evolution. This line of research is currently not so well connected to the field of complex systems as could be expected. The purpose of this paper is to provide an introduction to animal social networks for complex systems scientists and highlight areas of synergy. We believe that an increased integration of animal social networks with the interdisciplinary field of complex systems and networks would be beneficial for various reasons. Increased collaboration between researchers in this field and biologists studying animal social systems could be valuable in solving challenges that are of importance to animal social network research. Furthermore, animal social networks provide the opportunity to investigate hypotheses about complex systems across a range of natural real-world social systems. In this paper, we describe what animal social networks are and main research themes where they are studied; we give an overview of the methods commonly used to study animal social networks; we highlight challenges in the study of animal social networks where complex systems expertise may be particularly valuable; and we consider aspects of animal social networks that may be of particular interest to complex systems researchers. We hope that this will help to facilitate further interdisciplinary collaborations involving animal social networks, and further integration of these networks into the field of complex systems. △ Less

Submitted 19 February, 2021; v1 submitted 19 May, 2020; originally announced May 2020.

Comments: 15 pages, 2 figures

MolSSI and BioExcel Workflow Workshop 2018 Report

Authors: Levi N. Naden, Sam Ellis, Shantenu Jha

Abstract: Workflows in biomolecular science are very important as they are intricately intertwined with the scientific outcomes, as well as algorithmic and methodological innovations. The use and effectiveness of workflow tools to meet the needs of the biomolecular science community is varied. MolSSI co-organized a biomolecular workflows workshop in December 2018 with the goal of identifying specific softwa… ▽ More Workflows in biomolecular science are very important as they are intricately intertwined with the scientific outcomes, as well as algorithmic and methodological innovations. The use and effectiveness of workflow tools to meet the needs of the biomolecular science community is varied. MolSSI co-organized a biomolecular workflows workshop in December 2018 with the goal of identifying specific software gaps and opportunities for improved workflow practices. This report captures presentations and discussion from that workshop. The workshop participants were primary tools developers, along with "neutral observers" and some biomolecular domain scientists. After contextualizing and motivating the workshop, the report covers the existing roles and emerging trends in how workflow systems are utilized. A few recurring observations are presented as recommendations for improving the use and effectiveness of workflow tools. The tools presented are discussed in Appendix B. △ Less

Submitted 28 May, 2019; originally announced May 2019.

Comments: 13 pages, Workflow Developers, Workshop, Report

Robust Chemical Circuits

Authors: Samuel J. Ellis, Titus H. Klinge, James I. Lathrop

Abstract: We introduce a new motif for constructing robust digital logic circuits using input/output chemical reaction networks. These chemical circuits robustly handle perturbations in input signals, initial concentrations, rate constants, and measurements. In particular, we show that all combinatorial circuits and several sequential circuits enjoy this robustness. Our results complement existing literatur… ▽ More We introduce a new motif for constructing robust digital logic circuits using input/output chemical reaction networks. These chemical circuits robustly handle perturbations in input signals, initial concentrations, rate constants, and measurements. In particular, we show that all combinatorial circuits and several sequential circuits enjoy this robustness. Our results complement existing literature in the following three ways: (1) our logic gates read their inputs catalytically which make `fanout' gates unnecessary; (2) formal requirements and rigorous proofs of satisfaction are provided for each circuit; and (3) robustness of every circuit is closed under modular composition. △ Less

Submitted 21 August, 2018; originally announced August 2018.

The Effect of Focal Distance, Age, and Brightness on Near-Field Augmented Reality Depth Matching

Authors: Gujot Singh, Stephen R. Ellis, J. Edward Swan II

Abstract: Many augmented reality (AR) applications operate within near-field reaching distances, and require matching the depth of a virtual object with a real object. The accuracy of this matching was measured in three experiments, which examined the effect of focal distance, age, and brightness, within distances of 33.3 to 50 cm, using a custom-built AR haploscope. Experiment I examined the effect of foca… ▽ More Many augmented reality (AR) applications operate within near-field reaching distances, and require matching the depth of a virtual object with a real object. The accuracy of this matching was measured in three experiments, which examined the effect of focal distance, age, and brightness, within distances of 33.3 to 50 cm, using a custom-built AR haploscope. Experiment I examined the effect of focal demand, at the levels of collimated (infinite focal distance), consistent with other depth cues, and at the midpoint of reaching distance. Observers were too young to exhibit age-related reductions in accommodative ability. The depth matches of collimated targets were increasingly overestimated with increasing distance, consistent targets were slightly underestimated, and midpoint targets were accurately estimated. Experiment II replicated Experiment I, with older observers. Results were similar to Experiment I. Experiment III replicated Experiment I with dimmer targets, using young observers. Results were again consistent with Experiment I, except that both consistent and midpoint targets were accurately estimated. In all cases, collimated results were explained by a model, where the collimation biases the eyes' vergence angle outwards by a constant amount. Focal demand and brightness affect near-field AR depth matching, while age-related reductions in accommodative ability have no effect. △ Less

Submitted 30 November, 2017; originally announced December 2017.

Comments: 14 pages, 18 figures

Runtime Fault Detection in Programmed Molecular Systems

Authors: Samuel J. Ellis, Titus H. Klinge, James I. Lathrop, Jack H. Lutz, Robyn R. Lutz, Andrew S. Miner, Hugh D. Potter

Abstract: Watchdog timers are devices that are commonly used to monitor the health of safety-critical hardware and software systems. Their primary function is to raise an alarm if the monitored systems fail to emit periodic "heartbeats" that signal their well-being. In this paper we design and verify a molecular watchdog timer for monitoring the health of programmed molecular nanosystems. This raises new ch… ▽ More Watchdog timers are devices that are commonly used to monitor the health of safety-critical hardware and software systems. Their primary function is to raise an alarm if the monitored systems fail to emit periodic "heartbeats" that signal their well-being. In this paper we design and verify a molecular watchdog timer for monitoring the health of programmed molecular nanosystems. This raises new challenges because our molecular watchdog timer and the system that it monitors both operate in the probabilistic environment of chemical kinetics, where many failures are certain to occur and it is especially hard to detect the absence of a signal. Our molecular watchdog timer is the result of an incremental design process that uses goal-oriented requirements engineering, simulation, stochastic analysis, and software verification tools. We demonstrate the molecular watchdog's functionality by having it monitor a molecular oscillator. Both the molecular watchdog timer and the oscillator are implemented as chemical reaction networks, which are the current programming language of choice for many molecular programming applications. △ Less

Submitted 23 July, 2018; v1 submitted 25 October, 2017; originally announced October 2017.