-
Notifications
You must be signed in to change notification settings - Fork 241
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PyramidOMETiffWriter close()
duration scales as a (number of planes)^2
#4204
Comments
This issue has been mentioned on Image.sc Forum. There might be relevant details there: |
With a test based on https://github.com/ome/bioformats/blob/develop/components/formats-gpl/utils/MinimumPyramidWriter.java that writes one pyramid resolution: https://gist.github.com/melissalinkert/aaa9ac52c42a6b9c7596037cbadde4a3 I can see similar degradation of
I also see that data writing does not exactly scale linearly either, but obviously not as badly as Incidentally, with a similar test that writes multiple resolutions: https://gist.github.com/melissalinkert/ef601854820ee1a2ce970f09559a0cb0 I could easily reproduce #3907 using:
All four related issues (this, #3983, #3480, and now #3907) are now scheduled for 8.0.0, as discussed with @joshmoore and @sbesson. |
With the patch of @melissalinkert in this issue vanishes: in log scale: Raw perf data:
Code used (ijp-kheops lib): /*-
* #%L
* %%
* %%
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as
* published by the Free Software Foundation, either version 2 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public
* License along with this program. If not, see
* <https://www.gnu.org/licenses/gpl-2.0.html>.
* #L%
*/
import ch.epfl.biop.kheops.ometiff.OMETiffExporter;
import loci.common.DebugTools;
import net.imagej.ImageJ;
import net.imglib2.FinalInterval;
import net.imglib2.RandomAccessibleInterval;
import net.imglib2.position.FunctionRandomAccessible;
import net.imglib2.type.numeric.ARGBType;
import net.imglib2.view.Views;
import org.scijava.task.TaskService;
import java.time.Duration;
import java.time.Instant;
public class DemoExport {
public static void main( String[] args )
{
ImageJ ij = new ImageJ();
DebugTools.setRootLevel("Off");
ij.ui().showUI();
int sizeInPixelX = 512;
int sizeInPixelY = 512;
int tileSize = 512;
//int nT = 512;
for (int nT = 1; nT<=1024; nT*=2) {
final FunctionRandomAccessible<ARGBType> checkerboard = new FunctionRandomAccessible<>(
2,
(location, value) -> {
value.set(
ARGBType.rgba(location.getIntPosition(0), location.getIntPosition(1), location.getIntPosition(0) * 0, 255)
);
},
ARGBType::new);
RandomAccessibleInterval<ARGBType> img = Views.interval(checkerboard, new FinalInterval(new long[]{0, 0}, new long[]{sizeInPixelX - 1, sizeInPixelY - 1}));
DebugTools.setRootLevel("OFF");
try {
//String path = "C:\\Users\\nicol\\Desktop\\ometiff\\ntest-" + sizeInPixelX + "x"+sizeInPixelY+"px-" + tileSize + "tile.ome.tiff";
String path = "C:\\kheops\\test_x-" + sizeInPixelX + "_y-" + sizeInPixelY + "_tile-" + tileSize + "_nT-" + nT + ".ome.tiff";
System.out.println("Saving " + path);
Instant start = Instant.now();
OMETiffExporter.OMETiffExporterBuilder.Data.DataBuilder dataBuilder = OMETiffExporter.builder();
for (int t = 0; t < nT; t++) {
dataBuilder.putXYZRAI(0, t, img);
}
dataBuilder.defineMetaData("Image")
.imageName("Bob")
.voxelPhysicalSizeMicrometer(10, 10, 2)
.pixelsTimeIncrementInS(0.35)
.defineWriteOptions()
.tileSize(tileSize, tileSize)
.monitor(ij.get(TaskService.class))
.savePath(path)
.create()
.export();
Instant end = Instant.now();
System.out.println("Export time (ms) \t" + Duration.between(start, end).toMillis());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("done");
}
}
} (report made with the help of chatgpt) |
Thank you @NicoKiaru - this report is very helpful! |
Hello,
Original post: https://forum.image.sc/t/pyramidometiffwriter-really-bad-performance-when-closing-the-writer/95294
There is a (very big) performance issue when saving an OME TIFF file with
PyramidOMETiffWriter
as theclose()
method duration scales as the number of planes squared.I've tested to save a 512x512 pixel file on my computer and increase the number of timepoints by powers of 2 (nT). I've measured the time required to save the file and split the required time between what happens before calling close, and the close duration:
The data writing time scales linearly with the number of timepoints, which is what we expect. However the close time is multiplied by 4 each time the number of timepoints is multiplied by 2.
As a result, 'close' takes 90% of the total export time when there are 1024 timepoints.
I've narrowed the issue to the call of
TiffParser#getIFDOffsets
for all planes,getIFDOffsets
walking the whole file from the start. This scales very badly.PyramidOMETiffWriter#close
loops through all planes:bioformats/components/formats-bsd/src/loci/formats/out/PyramidOMETiffWriter.java
Line 125 in 7ca9980
TiffSaver#overwriteIFDValue
bioformats/components/formats-bsd/src/loci/formats/out/PyramidOMETiffWriter.java
Line 145 in 7ca9980
TiffParser#getIFDOffsets
bioformats/components/formats-bsd/src/loci/formats/tiff/TiffSaver.java
Line 804 in 7ca9980
TiffParser#getIFDOffsets
walks through the whole filebioformats/components/formats-bsd/src/loci/formats/tiff/TiffParser.java
Lines 362 to 363 in 7ca9980
Is there a way to fix this ?
This issue could be related to these other issues:
#3983
#3480
The text was updated successfully, but these errors were encountered: