Out of memory when invoking several times .hashes

[demanuel]

Hi,

I just got an error "Out of memory". The SQLite file is 368M.

Steps:

> raku
To exit type 'exit' or '^D'
> use DB::SQLite;
Nil
> my $dbh = DB::SQLite.new(filename => 'data/proxy.sqlite');
DB::SQLite.new(filename => "data/proxy.sqlite", busy-timeout => 10000, max-connections => 5, connections => Concurrent::Stack.new)
> my $result = $dbh.query('select * from requests limit 1');
DB::SQLite::Result.new(stmt => DB::SQLite::Native::Statement.new, count => 10, sth => DB::SQLite::Statement.new(stmt => DB::SQLite::Native::Statement.new, count => 10, db => DB::SQLite::Connection.new(conn => DB::SQLite::Native.new, owner => DB::SQLite.new(filename => "data/proxy.sqlite", busy-timeout => 10000, max-connections => 5, connections => Concurrent::Stack.new))), finish => Bool::True, keys-cache => Any)
> $result.hash
{cache => 86400, content_type => image/jpeg, host => images-webcams.windy.com, id => 1, method => GET, port => 443, protocol => HTTP/1.1, raw_response => Buf[uint8]:0x<48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D 0A 53 65 72 76 65 72 3A 20 6E 67 69 6E 78 2F 31 2E 31 30 2E 33 0D 0A 44 61 74 65 3A 20 53 75 6E 2C 20 33 31 20 4D 61 79 20 32 30 32 30 20 31 32 3A 30 36 3A 31 33 20 47 4D 54 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 70 65 3A 20 69 6D 61 67 65 2F 6A 70 65 67 ...>, status => 200 OK, uri => /16/1292677816/daylight/thumbnail/1292677816.jpg}
> $result.hashes
()
> say $_<content_type> for $result.hashes
image/jpeg
> say $_<content_type> for $result.hashes
image/jpeg
> say $_<content_type> for $result.hashes
image/jpeg
> say $_<content_type> for $result.hashes
image/jpeg
> say $_<content_type> for $result.hashes
out of memory
  in block  at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 42
  in method row at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 27
  in method pull-one at /home/demanuel/.raku/sources/51F6397FB5D02B2A0E15A62EC3ACF47669DBF3DC (DB::Result) line 21
  in block <unit> at <unknown file> line 1

> say $_<content_type> for $result.hashes
out of memory
  in block  at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 42
  in method row at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 27
  in method pull-one at /home/demanuel/.raku/sources/51F6397FB5D02B2A0E15A62EC3ACF47669DBF3DC (DB::Result) line 21
  in block <unit> at <unknown file> line 1

> 

[demanuel]

It also seems to not work reliably as some invocations returned correctly {count(1) => 475} and others returned wrongly {}

> my $result = $dbh.query('select count(1) from requests');
DB::SQLite::Result.new(stmt => DB::SQLite::Native::Statement.new, count => 1, sth => DB::SQLite::Statement.new(stmt => DB::SQLite::Native::Statement.new, count => 1, db => DB::SQLite::Connection.new(conn => DB::SQLite::Native.new, owner => DB::SQLite.new(filename => "data/proxy.sqlite", busy-timeout => 10000, max-connections => 5, connections => Concurrent::Stack.new))), finish => Bool::True, keys-cache => Any)
> $result.hash
{count(1) => 475}
> $result.hash
{}
> $result.hash
{count(1) => 475}
> $result.hash
{}
> $result.hash
out of memory
  in block  at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 42
  in method row at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 27
  in method hash at /home/demanuel/.raku/sources/51F6397FB5D02B2A0E15A62EC3ACF47669DBF3DC (DB::Result) line 68
  in block <unit> at <unknown file> line 1

>

[CurtTilmes]

hashes uses an iterator to make a Seq with a one time pass through the data. If you want to use it multiple times, assign it to a variable. Don't keep reusing the result object -- Just get out of it what you need and let it finish (which frees the memory). Exhausting the hashes iterator by reading into a variable will finish it automatically and free its memory.

my @hashes = $dbh.query('select * from requests limit 1').hashes;
say $_<content_type> for @hashes; 

BTW, you can see memory used by SQLite objects by printing DB::SQLite::Native.memory-used at any time. If you have hanging objects (could be cached connections or statements, or un-finished results) you will see how much memory they are holding by printing that.

[demanuel]

Is this applicable even when it's 1 element? I had the "limit 1" clause....

> say $_<content_type> for $result.hashes
image/jpeg
> DB::SQLite::Native.memory-used
325408
> say $_<content_type> for $result.hashes
image/jpeg
> DB::SQLite::Native.memory-used
325408
> say $_<content_type> for $result.hashes
image/jpeg
> DB::SQLite::Native.memory-used
325408
> say $_<content_type> for $result.hashes
image/jpeg
> DB::SQLite::Native.memory-used
162920
> say $_<content_type> for $result.hashes
out of memory
  in block  at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 42
  in method row at /home/demanuel/.raku/sources/3481BCAD53FC3338AC505479B91D3AD1F5BF3724 (DB::SQLite::Result) line 27
  in method pull-one at /home/demanuel/.raku/sources/51F6397FB5D02B2A0E15A62EC3ACF47669DBF3DC (DB::Result) line 21
  in block <unit> at <unknown file> line 1

>

Only when it decreased the error appeared. And what memory are we talking about? My system has 32GiB of memory. That value shown is not enough for getting an out of memory.

There is also the issue that if i remove the "limit 1" clause, it becames unusable:

To exit type 'exit' or '^D'
> use DB::SQLite;
Nil
> my $dbh = DB::SQLite.new(filename => 'data/proxy.sqlite');
DB::SQLite.new(filename => "data/proxy.sqlite", busy-timeout => 10000, max-connections => 5, connections => Concurrent::Stack.new)
> DB::SQLite::Native.memory-used
0
> DB::SQLite::Native.memory-used
0
> my $result = $dbh.query('select * from requests limit 1');
DB::SQLite::Result.new(stmt => DB::SQLite::Native::Statement.new, count => 10, sth => DB::SQLite::Statement.new(stmt => DB::SQLite::Native::Statement.new, count => 10, db => DB::SQLite::Connection.new(conn => DB::SQLite::Native.new, owner => DB::SQLite.new(filename => "data/proxy.sqlite", busy-timeout => 10000, max-connections => 5, connections => Concurrent::Stack.new))), finish => Bool::True, keys-cache => Any)
> DB::SQLite::Native.memory-used
162520
> my $ct= $result.hashes[0]<content_type>;
image/jpeg
> DB::SQLite::Native.memory-used
168552
> my $result = $dbh.query('select * from requests');
DB::SQLite::Result.new(stmt => DB::SQLite::Native::Statement.new, count => 10, sth => DB::SQLite::Statement.new(stmt => DB::SQLite::Native::Statement.new, count => 10, db => DB::SQLite::Connection.new(conn => DB::SQLite::Native.new, owner => DB::SQLite.new(filename => "data/proxy.sqlite", busy-timeout => 10000, max-connections => 5, connections => Concurrent::Stack.new))), finish => Bool::True, keys-cache => Any)
> DB::SQLite::Native.memory-used
331072
> my %ct= $result.hashes;

It is stuck here for at least 30 mins (until i ctrl-c).

[CurtTilmes]

I suspect the 'out of memory' is a spurious error. The Results object is a one-time use object. You get the results, you pull them out, then you don't use the object again. I'll try to add some more checks to the object to prevent its reuse, but the fact remains that you just can't do that. Decide what you want out of it (value/hash/array/arrays/hashes), then call it one time to get the result, then don't access the result any more.

my $ct= $result.hashes[0]<content_type>;

This says that you want a sequence of hashes, then you access the first one. The sequence then hangs around waiting for you to get the rest (doesn't matter if there is just one).

If you just want a single hash, just call hash:

my $ct = $result.hash<content_type>;

my %ct= $result.hashes;

hashes returns a sequence of hashes -- if you want just one, call hash:

my %ct = $result.hash;

[demanuel]

ok, got it.

Any reason why invoking .hashes being so slow?

[CurtTilmes]

Building each hash is slow. First it retrieves the array, then pairs the values with keys to make the hashes. If performance is a concern, always use arrays.