We propose a new framework for the analysis of current and future cosmological surveys, which combines perturbative methods (PT) on large scales with conditional simulation-based implicit inference (SBI) on small scales. This enables modeling of a wide range of statistics across all scales using only small-volume simulations, drastically reducing computational costs, and avoids the assumption of an explicit small-scale likelihood. As a proof-of-principle for this hybrid simulation-based inference (HySBI) approach, we apply it to dark matter density fields and constrain cosmological parameters using both the power spectrum and wavelet coefficients, finding promising results that significantly outperform classical PT methods. We additionally lay out a roadmap for the next steps necessary to implement HySBI on actual survey data, including consideration of bias, systematics, and customized simulations. Our approach provides a realistic way to scale SBI to future survey volumes, avoiding prohibitive computational costs.