Tags: minio/sio
Tags
Fix ReadAt with offset==EOF (#51) * Fix ReadAt with offset==EOF Fix a problem when a `ReadAt` requests a read at the offset that in plaintext corresponds exactly to a block starting at the EOF. We forward to where the block *would* be, thereby we never read the previous block header, and therefore do not get the "final" marker set We fix by reading (and skipping) the previous block if a request starts exactly at a block boundary to pick up the "final" marker. We calculate `t`, to be the number of full blocks to skip and `k` to be the number of bytes to skip before the response starts: ``` t := offset / int64(maxPayloadSize) k := offset % int64(maxPayloadSize) ``` We add the following if we start at a block boundary after the first block: ``` if offset > 0 && k == 0 { k = maxPayloadSize t-- } ``` This will read the previous block to read the "final" flag and skip its content.
Add buffer pool and stateful errors (#42) * Make Reader and Close on Writers errors stateful, so they can't be silently ignored. * Add buffer pool, to help small operations. * Make benchmarks parallel, so they can be controlled with `-cpu` param. Stateful errors are required for buffers and are good practice anyway. ``` benchmark old ns/op new ns/op delta BenchmarkEncryptReader_8KB 7040 1754 -75.09% BenchmarkEncryptReader_8KB-32 5711 564 -90.13% BenchmarkEncryptReader_64KB 23739 17406 -26.68% BenchmarkEncryptReader_64KB-32 8358 1680 -79.90% BenchmarkEncryptReader_512KB 143045 135148 -5.52% BenchmarkEncryptReader_512KB-32 17855 9144 -48.79% BenchmarkEncryptReader_1MB 274863 266411 -3.07% BenchmarkEncryptReader_1MB-32 28966 17709 -38.86% BenchmarkDecryptReader_8KB 8390 2036 -75.73% BenchmarkDecryptReader_8KB-32 5852 700 -88.03% BenchmarkDecryptReader_64KB 24062 17358 -27.86% BenchmarkDecryptReader_64KB-32 6742 2038 -69.77% BenchmarkDecryptReader_512KB 136199 128197 -5.88% BenchmarkDecryptReader_512KB-32 17214 9722 -43.52% BenchmarkDecryptReader_1MB 261589 256942 -1.78% BenchmarkDecryptReader_1MB-32 29842 18367 -38.45% BenchmarkDecryptReaderAt_8KB 16163 3108 -80.77% BenchmarkDecryptReaderAt_8KB-32 11139 1011 -90.92% BenchmarkDecryptReaderAt_64KB 48030 33390 -30.48% BenchmarkDecryptReaderAt_64KB-32 12682 2859 -77.46% BenchmarkDecryptReaderAt_512KB 135685 122129 -9.99% BenchmarkDecryptReaderAt_512KB-32 21756 8825 -59.44% BenchmarkDecryptReaderAt_1MB 253436 241940 -4.54% BenchmarkDecryptReaderAt_1MB-32 29090 17011 -41.52% BenchmarkEncryptWriter_8KB 8110 1721 -78.78% BenchmarkEncryptWriter_8KB-32 5590 460 -91.77% BenchmarkEncryptWriter_64KB 23016 17731 -22.96% BenchmarkEncryptWriter_64KB-32 6519 1212 -81.41% BenchmarkEncryptWriter_512KB 128791 125068 -2.89% BenchmarkEncryptWriter_512KB-32 9761 8861 -9.22% BenchmarkEncryptWriter_1MB 248845 244758 -1.64% BenchmarkEncryptWriter_1MB-32 18768 17645 -5.98% BenchmarkDecryptWriter_8KB 10045 1721 -82.87% BenchmarkDecryptWriter_8KB-32 10071 612 -93.93% BenchmarkDecryptWriter_64KB 25237 16298 -35.42% BenchmarkDecryptWriter_64KB-32 7284 1506 -79.32% BenchmarkDecryptWriter_512KB 130734 122230 -6.50% BenchmarkDecryptWriter_512KB-32 14006 8581 -38.73% BenchmarkDecryptWriter_1MB 249146 240821 -3.34% BenchmarkDecryptWriter_1MB-32 22672 17055 -24.78% benchmark old MB/s new MB/s speedup BenchmarkEncryptReader_8KB 145.45 583.65 4.01x BenchmarkEncryptReader_8KB-32 179.29 1816.31 10.13x BenchmarkEncryptReader_64KB 2760.74 3765.09 1.36x BenchmarkEncryptReader_64KB-32 7841.21 38999.69 4.97x BenchmarkEncryptReader_512KB 3665.21 3879.36 1.06x BenchmarkEncryptReader_512KB-32 29363.77 57335.81 1.95x BenchmarkEncryptReader_1MB 3814.91 3935.93 1.03x BenchmarkEncryptReader_1MB-32 36200.60 59211.25 1.64x BenchmarkDecryptReader_8KB 122.05 503.07 4.12x BenchmarkDecryptReader_8KB-32 174.98 1461.79 8.35x BenchmarkDecryptReader_64KB 2723.60 3775.56 1.39x BenchmarkDecryptReader_64KB-32 9720.77 32154.53 3.31x BenchmarkDecryptReader_512KB 3849.43 4089.72 1.06x BenchmarkDecryptReader_512KB-32 30457.05 53925.25 1.77x BenchmarkDecryptReader_1MB 4008.49 4080.99 1.02x BenchmarkDecryptReader_1MB-32 35137.48 57088.75 1.62x BenchmarkDecryptReaderAt_8KB 63.36 329.47 5.20x BenchmarkDecryptReaderAt_8KB-32 91.93 1012.87 11.02x BenchmarkDecryptReaderAt_64KB 1364.47 1962.73 1.44x BenchmarkDecryptReaderAt_64KB-32 5167.56 22921.02 4.44x BenchmarkDecryptReaderAt_512KB 3864.01 4292.92 1.11x BenchmarkDecryptReaderAt_512KB-32 24098.26 59407.79 2.47x BenchmarkDecryptReaderAt_1MB 4137.44 4334.04 1.05x BenchmarkDecryptReaderAt_1MB-32 36046.33 61642.61 1.71x BenchmarkEncryptWriter_8KB 126.27 594.86 4.71x BenchmarkEncryptWriter_8KB-32 183.17 2226.33 12.15x BenchmarkEncryptWriter_64KB 2847.45 3696.05 1.30x BenchmarkEncryptWriter_64KB-32 10052.84 54070.81 5.38x BenchmarkEncryptWriter_512KB 4070.85 4192.01 1.03x BenchmarkEncryptWriter_512KB-32 53709.97 59168.98 1.10x BenchmarkEncryptWriter_1MB 4213.78 4284.14 1.02x BenchmarkEncryptWriter_1MB-32 55871.68 59426.78 1.06x BenchmarkDecryptWriter_8KB 101.94 594.91 5.84x BenchmarkDecryptWriter_8KB-32 101.68 1673.98 16.46x BenchmarkDecryptWriter_64KB 2596.84 4021.05 1.55x BenchmarkDecryptWriter_64KB-32 8996.95 43507.52 4.84x BenchmarkDecryptWriter_512KB 4010.34 4289.34 1.07x BenchmarkDecryptWriter_512KB-32 37431.90 61100.09 1.63x BenchmarkDecryptWriter_1MB 4208.68 4354.17 1.03x BenchmarkDecryptWriter_1MB-32 46249.03 61480.32 1.33x benchmark old allocs new allocs delta BenchmarkEncryptReader_8KB 12 11 -8.33% BenchmarkEncryptReader_8KB-32 12 11 -8.33% BenchmarkEncryptReader_64KB 12 11 -8.33% BenchmarkEncryptReader_64KB-32 12 11 -8.33% BenchmarkEncryptReader_512KB 19 18 -5.26% BenchmarkEncryptReader_512KB-32 19 18 -5.26% BenchmarkEncryptReader_1MB 27 26 -3.70% BenchmarkEncryptReader_1MB-32 27 26 -3.70% BenchmarkDecryptReader_8KB 18 17 -5.56% BenchmarkDecryptReader_8KB-32 18 17 -5.56% BenchmarkDecryptReader_64KB 18 17 -5.56% BenchmarkDecryptReader_64KB-32 18 17 -5.56% BenchmarkDecryptReader_512KB 25 24 -4.00% BenchmarkDecryptReader_512KB-32 25 24 -4.00% BenchmarkDecryptReader_1MB 33 32 -3.03% BenchmarkDecryptReader_1MB-32 33 32 -3.03% BenchmarkDecryptReaderAt_8KB 24 22 -8.33% BenchmarkDecryptReaderAt_8KB-32 24 22 -8.33% BenchmarkDecryptReaderAt_64KB 24 22 -8.33% BenchmarkDecryptReaderAt_64KB-32 24 22 -8.33% BenchmarkDecryptReaderAt_512KB 29 27 -6.90% BenchmarkDecryptReaderAt_512KB-32 29 27 -6.90% BenchmarkDecryptReaderAt_1MB 37 35 -5.41% BenchmarkDecryptReaderAt_1MB-32 37 35 -5.41% BenchmarkEncryptWriter_8KB 12 11 -8.33% BenchmarkEncryptWriter_8KB-32 12 11 -8.33% BenchmarkEncryptWriter_64KB 12 11 -8.33% BenchmarkEncryptWriter_64KB-32 12 11 -8.33% BenchmarkEncryptWriter_512KB 19 18 -5.26% BenchmarkEncryptWriter_512KB-32 19 18 -5.26% BenchmarkEncryptWriter_1MB 27 26 -3.70% BenchmarkEncryptWriter_1MB-32 27 26 -3.70% BenchmarkDecryptWriter_8KB 14 13 -7.14% BenchmarkDecryptWriter_8KB-32 14 13 -7.14% BenchmarkDecryptWriter_64KB 14 13 -7.14% BenchmarkDecryptWriter_64KB-32 14 13 -7.14% BenchmarkDecryptWriter_512KB 21 20 -4.76% BenchmarkDecryptWriter_512KB-32 21 20 -4.76% BenchmarkDecryptWriter_1MB 29 28 -3.45% BenchmarkDecryptWriter_1MB-32 29 28 -3.45% benchmark old bytes new bytes delta BenchmarkEncryptReader_8KB 74877 1144 -98.47% BenchmarkEncryptReader_8KB-32 75206 1549 -97.94% BenchmarkEncryptReader_64KB 74877 1144 -98.47% BenchmarkEncryptReader_64KB-32 75646 1705 -97.75% BenchmarkEncryptReader_512KB 74990 1256 -98.33% BenchmarkEncryptReader_512KB-32 75902 1764 -97.68% BenchmarkEncryptReader_1MB 75117 1384 -98.16% BenchmarkEncryptReader_1MB-32 75868 1712 -97.74% BenchmarkDecryptReader_8KB 75125 1392 -98.15% BenchmarkDecryptReader_8KB-32 75171 1908 -97.46% BenchmarkDecryptReader_64KB 75126 1392 -98.15% BenchmarkDecryptReader_64KB-32 75182 2012 -97.32% BenchmarkDecryptReader_512KB 75237 1504 -98.00% BenchmarkDecryptReader_512KB-32 75347 1709 -97.73% BenchmarkDecryptReader_1MB 75366 1633 -97.83% BenchmarkDecryptReader_1MB-32 75505 1809 -97.60% BenchmarkDecryptReaderAt_8KB 149123 1656 -98.89% BenchmarkDecryptReaderAt_8KB-32 149729 2544 -98.30% BenchmarkDecryptReaderAt_64KB 149126 1657 -98.89% BenchmarkDecryptReaderAt_64KB-32 149804 2221 -98.52% BenchmarkDecryptReaderAt_512KB 149255 1784 -98.80% BenchmarkDecryptReaderAt_512KB-32 149511 1935 -98.71% BenchmarkDecryptReaderAt_1MB 149542 2074 -98.61% BenchmarkDecryptReaderAt_1MB-32 150141 2381 -98.41% BenchmarkEncryptWriter_8KB 74843 1112 -98.51% BenchmarkEncryptWriter_8KB-32 74854 1354 -98.19% BenchmarkEncryptWriter_64KB 74883 1144 -98.47% BenchmarkEncryptWriter_64KB-32 75237 1206 -98.40% BenchmarkEncryptWriter_512KB 77223 3423 -95.57% BenchmarkEncryptWriter_512KB-32 80640 6186 -92.33% BenchmarkEncryptWriter_1MB 85695 11542 -86.53% BenchmarkEncryptWriter_1MB-32 101491 25308 -75.06% BenchmarkDecryptWriter_8KB 75002 1272 -98.30% BenchmarkDecryptWriter_8KB-32 75029 1720 -97.71% BenchmarkDecryptWriter_64KB 75004 1272 -98.30% BenchmarkDecryptWriter_64KB-32 75028 1662 -97.78% BenchmarkDecryptWriter_512KB 75175 1438 -98.09% BenchmarkDecryptWriter_512KB-32 75304 1561 -97.93% BenchmarkDecryptWriter_1MB 75468 1730 -97.71% BenchmarkDecryptWriter_1MB-32 75857 2029 -97.33% ```