research-article

Performance comprehension at WiredTiger

Authors:

Alexandra Fedorova,

Ivan Beschastnikh,

Augustine Wong,

Svetozar Miucin,

Louis YeAuthors Info & Claims

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 83 - 94

https://doi.org/10.1145/3236024.3236081

Published: 26 October 2018 Publication History

Abstract

Software debugging is a time-consuming and challenging process. Supporting debugging has been a focus of the software engineering field since its inception with numerous empirical studies, theories, and tools to support developers in this task. Performance bugs and performance debugging is a sub-genre of debugging that has received less attention.

In this paper we contribute an empirical case study of performance bug diagnosis in the WiredTiger project, the default database engine behind MongoDB. We perform an in-depth analysis of 44 Jira tickets documenting WiredTiger performance-related issues. We investigate how developers diagnose performance bugs: what information they collect, what tools they use, and what processes they follow. Our findings show that developers spend the majority of their performance debugging time chasing outlier events, such as latency spikes and throughput drops. Yet, they are not properly supported by existing performance debugging tools in this task. We also observe that developers often use tools without knowing in advance whether the obtained information will be relevant to debugging the problem. Therefore, we believe developers can benefit from tools that can be used for unstructured exploration of performance data, rather than for answering specific questions.

References

[1]

Datacenter Computers: Modern Challenges in CPU Design. https://www.pdl.cmu. edu/SDI/2015/slides/DatacenterComputers.pdf.

[2]

DTrace. https://dtrace.org/blogs/about.

[3]

Intel Processor Trace. https://software.intel.com/en-us/blogs/2013/09/18/ processor-tracing.

[4]

Jenkins. https://jenkins.io/.

[5]

Poor Man’s Profiler. https://poormansprofiler.org.

[6]

RotateRight Zoom. https://en.wikipedia.org/wiki/RotateRight_Zoom.

[7]

The DINAMITE Toolkit. https://dinamite-toolkit.github.io.

[8]

WiredTiger operation tracking tutorial. https://github.com/wiredtiger/ wiredtiger/wiki/WiredTiger-operation-tracking.

[9]

J. Aranda and G. Venolia. The Secret Life of Bugs: Going Past the Errors and Omissions in Software Repositories. In Proceedings of the International Conference on Software Engineering, ICSE, 2009.

Digital Library

[10]

S. Baltes, O. Moseler, F. Beck, and S. Diehl. Navigate, Understand, Communicate: How Developers Locate Performance Bugs. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement, ESEM, 2015.

[11]

D. M. Berris, A. Veitch, N. Heintze, E. Anderson, and N. Wang. XRay: A Function Call Tracing System. Technical report, 2016. A white paper on XRay, a function call tracing system developed at Google.

[12]

J. Burnim, S. Juvekar, and K. Sen. WISE: Automated Test Generation for Worstcase Complexity. In Proceedings of the International Conference on Software Engineering, ICSE, 2009.

Digital Library

[13]

B. Chen, Y. Liu, and W. Le. Generating Performance Distributions via Probabilistic Symbolic Execution. In Proceedings of the International Conference on Software Engineering, ICSE, 2016.

Digital Library

[14]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC, 2010.

Digital Library

[15]

J. Corbin and A. Strauss. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, Inc., 3rd edition, 2008.

[16]

B. Cornelissen, D. Holten, A. Zaidman, J. J. Van Wijk, and A. Van Deursen. Understanding Execution Traces Using Massive Sequence and Circular Bundle Views. In Proceedings of the International Conference on Program Comprehension, ICPC, 2007.

Digital Library

[17]

W. De Pauw and S. Heisig. Zinsight: a Visual and Analytic Environment for Exploring Large Event Traces. In Proceedings of the International Symposium on Software Visualization, SOFTVIS, 2010.

Digital Library

[18]

J. Dean and L. A. Barroso. The Tail at Scale. Commun. ACM, 56(2):74–80, Feb. 2013.

Digital Library

[19]

K. C. Foo, Z. M. J. Jiang, B. Adams, A. E. Hassan, Y. Zou, and P. Flora. An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments. In Proceedings of the International Conference on Software Engineering, ICSE, 2015.

Digital Library

[20]

B. Glaser and A. Strauss. The Discovery of Grounded Theory: Strategies for Qualitative Research. Observations (Chicago, Ill.). Aldine de Gruyter, 1967.

[21]

G. Gousios, M. Pinzger, and A. v. Deursen. An Exploratory Study of the Pull-based Software Development Model. In Proceedings of the International Conference on Software Engineering, ICSE, 2014.

Digital Library

[22]

K. E. Isaacs, P.-T. Bremer, I. Jusufi, T. Gamblin, A. Bhatele, M. Schulz, and B. Hamann. Combing the Communication Hairball: Visualizing Parallel Execution Traces Using Logical Time. In IEEE Transactions on Visualization and Computer Graphics, 2014.

[23]

G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and Detecting Real-world Performance Bugs. In Proceedings of the Conference on Programming Language Design and Implementation, PLDI, 2012.

Digital Library

[24]

B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs? In Proceedings of the International Conference on Software Engineering, ICSE, 2013.

Digital Library

[25]

M. Jovic, A. Adamoli, and M. Hauswirth. Catch me if you can: performance bug detection in the wild. In Proceedings of the International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA, 2011.

Digital Library

[26]

B. Karran, J. Trümper, and J. Döllner. SYNCTRACE: Visual Thread-Interplay Analysis. In Proceedings of the Working Conference on Software Visualization, VISSOFT, 2013.

[27]

A. J. Ko, R. DeLine, and G. Venolia. Information Needs in Collocated Software Development Teams. In Proceedings of the International Conference on Software Engineering, ICSE, 2007.

Digital Library

[28]

T. D. LaToza and B. A. Myers. Developers Ask Reachability Questions. In Proceedings of the International Conference on Software Engineering, ICSE, 2010.

Digital Library

[29]

Y. Liu, C. Xu, and S.-C. Cheung. Characterizing and Detecting Performance Bugs for Smartphone Applications. In Proceedings of the International Conference on Software Engineering, ICSE, 2014.

Digital Library

[30]

Michael Cahill. A Technical Introduction to WiredTiger. https://www.slideshare. net/mongodb/mongo-db-wiredtigerwebinar.

[31]

MongoDB. MongoDB Acquires WiredTiger Inc. https://www.mongodb.com/ press/wired-tiger.

[32]

T. H. D. Nguyen, M. Nagappan, A. E. Hassan, M. Nasser, and P. Flora. An Industrial Case Study of Automatically Identifying Performance Regression-causes. In Proceedings of the Working Conference on Mining Software Repositories, MSR, 2014.

Digital Library

[33]

A. Nistor, T. Jiang, and L. Tan. Discovering, Reporting, and Fixing Performance Bugs. In Proceedings of the Working Conference on Mining Software Repositories, MSR, 2013.

Digital Library

[34]

A. Nistor, L. Song, D. Marinov, and S. Lu. Toddler: Detecting Performance Problems via Similar Memory-access Patterns. In Proceedings of the International Conference on Software Engineering, ICSE, 2013.

Digital Library

[35]

J. Ousterhout. Always Measure One Level Deeper. Commun. ACM, 61(7):74–83, June 2018.

Digital Library

[36]

R. Padhye and K. Sen. Travioli: A Dynamic Analysis for Detecting Data-structure Traversals. In Proceedings of the International Conference on Software Engineering, ICSE, 2017.

Digital Library

[37]

M. Pradel, M. Huggler, and T. R. Gross. Performance Regression Testing of Concurrent Classes. In Proceedings of the International Symposium on Software Testing and Analysis, ISSTA, 2014.

Digital Library

[38]

V. Singh, R. Gupta, and I. Neamtiu. MG++: Memory graphs for analyzing dynamic data structures. In Proceedings of the International Conference on Software Analysis, Evolution and Reengineering, SANER, 2015.

[39]

L. Song and S. Lu. Statistical Debugging for Real-world Performance Problems. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA, pages 561–578, 2014.

Digital Library

[40]

J. Trümper, J. Bohnet, and J. Döllner. Understanding Complex Multithreaded Software Systems by using Trace Visualization. In Proceedings of the International Symposium on Software Visualization, SOFTVIS, 2010.

Digital Library

[41]

J. Trümper, J. Döllner, and A. Telea. Multiscale Visual Comparison of Execution Traces. In Proceedings of the International Conference on Program Comprehension, ICPC, 2013.

[42]

WiredTiger. WiredTiger GitHub. https://github.com/wiredtiger/wiredtiger.

[43]

WiredTiger. WiredTiger Jira. https://jira.mongodb.org/projects/WT/issues.

[44]

Y. Wu, R. H. Yap, and F. Halim. Visualizing Windows System Traces. In Proceedings of the International Symposium on Software Visualization, SOFTVIS, 2010.

Digital Library

[45]

D. Yuan, S. Park, and Y. Zhou. Characterizing logging practices in open-source software. In Proceedings of the International Conference on Software Engineering, ICSE, 2012.

Digital Library

[46]

S. Zaman, B. Adams, and A. E. Hassan. A Qualitative Study on Performance Bugs. In Proceedings of the IEEE Working Conference on Mining Software Repositories, MSR, 2012.

Digital Library

[47]

X. Zhao, K. Rodrigues, Y. Luo, M. Stumm, D. Yuan, and Y. Zhou. Log20: Fully Automated Optimal Placement of Log Printing Statements Under Specified Overhead Threshold. In Proceedings of the Symposium on Operating Systems Principles, SOSP, 2017.

Digital Library

Cited By

Wong ABucci PBeschastnikh IFedorova A(2024)Making Sense of Multi-threaded Application Performance at Scale with NonSequiturProceedings of the ACM on Programming Languages10.1145/36897938:OOPSLA2(2325-2354)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689793
Kadekodi RKadekodi SPonnapalli SShirwadkar HGanger GKolli AChidambaram V(2021)WineFSProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles10.1145/3477132.3483567(804-818)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3477132.3483567
Sanchez ADelgado-Perez PMedina-Bulo ISegura S(2020)TANDEM: A Taxonomy and a Dataset of Real-World Performance BugsIEEE Access10.1109/ACCESS.2020.30009288(107214-107228)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3000928
Show More Cited By

Index Terms

Performance comprehension at WiredTiger
1. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development
      1. Programming teams
    2. Software development process management
      1. Software development methods
  2. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Performance prediction for portable parallel execution on MIMD architectures
IPPS '95: Proceedings of the 9th International Symposium on Parallel Processing

Performance debugging and prediction for parallel systems is a difficult problem. The difficulties in identifying performance bottlenecks stem from the need for an intimate understanding of the underlying architecture. It has been recognized that ...
Performance debugging for distributed systems of black boxes
SOSP '03

Many interesting large-scale systems are distributed systems of multiple communicating components. Such systems can be very hard to debug, especially when they exhibit poor performance. The problem becomes much harder when systems are composed of "black-...
Performance debugging based on scalability analysis
FRONTIERS '95: Proceedings of the Fifth Symposium on the Frontiers of Massively Parallel Computation (Frontiers'95)

This paper presents scalability as a basis for profiling and performance debugging of parallel programs, as only the purely scalable code runs efficiently in parallel. The approach is based on separating scalable and various kinds of non-scalable parts ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

October 2018

987 pages

ISBN:9781450355735

DOI:10.1145/3236024

General Chair:
Gary T. Leavens
University of Central Florida, USA
,
Program Chairs:
Alessandro Garcia
PUC-Rio, Brazil
,
Corina S. Păsăreanu
NASA Ames Research Center, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '18

Sponsor:

SIGSOFT

ESEC/FSE '18: 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 4 - 9, 2018

FL, Lake Buena Vista, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
304
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wong ABucci PBeschastnikh IFedorova A(2024)Making Sense of Multi-threaded Application Performance at Scale with NonSequiturProceedings of the ACM on Programming Languages10.1145/36897938:OOPSLA2(2325-2354)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689793
Kadekodi RKadekodi SPonnapalli SShirwadkar HGanger GKolli AChidambaram V(2021)WineFSProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles10.1145/3477132.3483567(804-818)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3477132.3483567
Sanchez ADelgado-Perez PMedina-Bulo ISegura S(2020)TANDEM: A Taxonomy and a Dataset of Real-World Performance BugsIEEE Access10.1109/ACCESS.2020.30009288(107214-107228)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3000928
Pasquier TEyers DSeltzer M(2019)From Here to ProvtopiaHeterogeneous Data Management, Polystores, and Analytics for Healthcare10.1007/978-3-030-33752-0_4(54-67)Online publication date: 23-Oct-2019
https://doi.org/10.1007/978-3-030-33752-0_4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents