Fix jq load_probabilities precision error using python #621
Conversation
`jq` has a precision bug while loading probabilities with big vids:
```
$ printf '1 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294272 200 300
$ printf '2 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294274 200 300
$ printf '3 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294276 200 300
$ printf '4 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294276 200 300
$ printf '5 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294276 200 300
$ printf '6 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294278 200 300
$ printf '7 200 300\n' | jq -R -r --argjson SHARD_BASE $((37 << 48)) '
split(" ") |
[ (.[0] | tonumber + $SHARD_BASE | tostring)
, .[1:][] ] | join("\t")'
10414574138294280 200 300
```
this is a known issue and shockingly there seems no good way in jq to support better precision:
jqlang/jq#369
We've tried using awk but versioning seems a problem. Here we just try to use python3 to do the job.
Just used a poor
| split(\" \") | | ||
| [ (.[0] | tonumber + $SHARD_BASE | tostring) | ||
| , .[1:][] ] | join(\"\\t\")' | | ||
| python3 -c " |
There was a problem hiding this comment.
I think you need to escape the double quote (with backslash). Travis is complaining about jq compilation.
Also, @netj should we just use python here -- which should be compatible with both 2 and 3.
@zifeishan let's do some perf test for this?
There was a problem hiding this comment.
I couldn't resist doing a benchmark notebook
and the winner is.... Perl!
jq was actually the slowest, ~7x slower than Perl and 3x than AWK. Python is ~1.7x, AWK is ~2.2x.
jq and AWK uses little memory ~10MB, ~1/3 of Python (35-45MB), but so does Perl (14MB).
So Perl is a clear winner.
There was a problem hiding this comment.
If perl can handle all 2^64 numbers and that's not subject to version differences on different OSes, we can go with perl.
jqhas a precision bug while loading probabilities with big vids, due to a known issue that it's using double floating numbers to store integers. There seems no good way in jq to support better precision:jqlang/jq#369
We've tried using awk with --bignum but versioning seems a problem.
Here we just try to use python3 to do the job.