Performance Issues for File Reading

[mayuehappy]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

I found that in the implementation of cloud_env_impl.NewRandomAccessFile, if we has local SstFileCache. When the local file does not exist, it needs to wait until the file is downloaded to the local. This process may be slower when the file is large.
Can we directly Random Access from S3 File, then asynchronously download the file and update the cache, after that we can replace the RandomAccessFile with the local one.

[githubzilla]

I guess if you disable sst_file_cache in CloudFileSystemOptions, it will directly randomly read S3 file.