Dou et al., 2023 - Google Patents

TurBO: A cost-efficient configuration-based auto-tuning approach for cluster-based big data frameworks

Dou et al., 2023

Document ID
7283273284763743357
Author
Dou H
Zhang L
Zhang Y
Chen P
Zheng Z
Publication year
Publication venue
Journal of Parallel and Distributed Computing

External Links

Snippet

Big data processing frameworks such as Spark usually provide a large number of performance-related configuration parameters, how to auto-tune these parameters for a better performance has been a hot issue in academia as well as industry for years. Through …
Continue reading at www.sciencedirect.com (other versions)

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30386—Retrieval requests
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30289—Database design, administration or maintenance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for programme control, e.g. control unit
    • G06F9/06—Arrangements for programme control, e.g. control unit using stored programme, i.e. using internal store of processing equipment to receive and retain programme
    • G06F9/46—Multiprogramming arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N99/00—Subject matter not provided for in other groups of this subclass
    • G06N99/005—Learning machines, i.e. computer in which a programme is changed according to experience gained by the machine itself during a complete run
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/50—Computer-aided design
    • G06F17/5009—Computer-aided design using simulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformations of program code
    • G06F8/41—Compilation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06Q—DATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management, e.g. organising, planning, scheduling or allocating time, human or machine resources; Enterprise planning; Organisational models
    • G06Q10/063—Operations research or analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computer systems utilising knowledge based models
    • G06N5/02—Knowledge representation
    • G06N5/022—Knowledge engineering, knowledge acquisition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computer systems based on specific mathematical models
    • G06N7/005—Probabilistic networks

Similar Documents

Publication Publication Date Title
Salloum et al. Big data analytics on Apache Spark
Marathe et al. Performance modeling under resource constraints using deep transfer learning
Siegmund et al. SPL Conqueror: Toward optimization of non-functional properties in software product lines
Chen et al. Temporal dependency-based checkpoint selection for dynamic verification of temporal constraints in scientific workflow systems
Cheng et al. Efficient performance prediction for apache spark
US10146531B2 (en) Method and apparatus for generating a refactored code
Singhal et al. Performance assurance model for applications on SPARK platform
Bei et al. Configuring in-memory cluster computing using random forest
Laptev et al. Very fast estimation for result and accuracy of big data analytics: The EARL system
Prats et al. You only run once: spark auto-tuning from a single run
Fekry et al. Tuneful: An online significance-aware configuration tuner for big data analytics
Krzywaniak et al. Dynamic GPU power capping with online performance tracing for energy efficient GPU computing using DEPO tool
US11829842B2 (en) Enhanced quantum circuit execution in a quantum service
Li et al. Towards general and efficient online tuning for spark
Dou et al. TurBO: A cost-efficient configuration-based auto-tuning approach for cluster-based big data frameworks
Rodrigues et al. Screening hardware and volume factors in distributed machine learning algorithms on spark: A design of experiments (doe) based approach
Qin et al. Auxiliary Gibbs Sampling for Inference in Piecewise-Constant Conditional Intensity Models.
Sindhu et al. Workload characterization and synthesis for cloud using generative stochastic processes
Rejitha et al. Energy prediction of CUDA application instances using dynamic regression models
Hao et al. Predicting QoS of virtual machines via Bayesian network with XGboost-induced classes
Chen et al. SimCost: cost-effective resource provision prediction and recommendation for spark workloads
Rouff et al. The AdaptiV approach to verification of adaptive systems
Choi et al. Optimizing Numerical Weather Prediction Model Performance using Machine Learning Techniques
Fasi et al. Challenges in computing matrix functions
Sharma et al. An empirical evaluation of cross project priority prediction