check is streamable

[AlexTMallen]

This also removes hard-coded checks for datasets

[norabelrose]

This seems to just hang when you try elicit on imdb with --stream on

[AlexTMallen]

This seems to just hang when you try elicit on imdb with --stream on

The hanging is caused by it streaming 750 examples to check if they all have the same label. I found that streaming the data necessarily takes this much time (in my case, about 8 seconds). This should only impact performance when it throws an Exception for the dataset not being streamable.

[norabelrose]

he hanging is caused by it streaming 750 examples to check if they all have the same label. I found that streaming the data necessarily takes this much time (in my case, about 8 seconds). This should only impact performance when it throws an Exception for the dataset not being streamable.

It's possible that I'm misremembering but I do distinctly remember actually going through with a print statement to see what was going on and it like, stayed frozen even after it had hit the appropriate number of examples

[AlexTMallen]

It's possible that I'm misremembering but I do distinctly remember actually going through with a print statement to see what was going on and it like, stayed frozen even after it had hit the appropriate number of examples

I can't replicate this when I use print statements. If you're using max_examples / world_size less than 100, it checks the first 100 streaming examples as a minimum. As soon as it reaches the appropriate iteration it raises an exception for me.

[CLAassistant]

All committers have signed the CLA.

[norabelrose]

We don't even support streaming datasets anymore