Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add the ability to call methods on the underlying JDBC Statement eg setFetchSize #437

Open
googley42 opened this issue Apr 24, 2021 · 0 comments

Comments

@googley42
Copy link

googley42 commented Apr 24, 2021

The automatic behaviour for drivers such as MySql is to read all the records of the resultset into memory. For very large resultsets this is undesirable and will blow memory client side. One way to control this is to manipulate the jdbc Statement object see this SO post
Eg statement.fetchSize = Integer.MIN_VALUE will give a hint to the driver to emit one row at a time.

So you would end up with code that is equivalent to this hard coded example

    override def readOn[A](read: Read[A], conn: Connection): Stream[Exception, A] =
      Stream.unwrap {
        blocking.effectBlocking {
          val schema = getColumns(read).zipWithIndex.map { case (value, index) =>
            (value, index + 1)
          } // SQL is 1-based indexing

          val query = renderRead(read)

          val statement = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)
          statement.setFetchDirection(ResultSet.FETCH_FORWARD)
          statement.setFetchSize(Int.MinValue)
     // etc etc   

I would normally pick this up but I am a bit snowed under ATM.

Actually as readOn returns a ZStream maybe we can get away with just hard coding the above for now for a quick win? - I can certainly create a PR for this simple defaulting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant