Empty output files when running under Grunt

[vicchi]

I'm trying to produce a Grunt plugin that wraps ogr2ogr functionality so I can use it without additional coding in my workflow. But I'm getting empty output files created, with no apparent errors or warnings.

As a standalone (and shabby) piece of code, the following works fine ...

var ogr2ogr = require('ogr2ogr');

var src = 'test/data/src/ne_110m_admin_0_sovereignty.shp';
var dest = 'tmp/ne_110m_admin_0_sovereignty.geojson';
var format = 'GeoJSON';

var ogr = ogr2ogr(src);
ogr.on('ogrinfo', console.error);
ogr.on('error', console.error);
ogr.format(format);
var output = ogr.stream();
output.pipe(fs.createWriteStream(dest));

However, the same code when invoked from within Grunt's environment produces no errors or warnings and while an output file is produced, it's empty.

'use strict';

module.exports = function(grunt) {
    var ogr2ogr = require('ogr2ogr');
    var fs = require('fs');
    var path = require('path');

    grunt.registerMultiTask('ogr2ogr', 'ogr2ogr wrapper task for Grunt', function() {
        // Merge task-specific and/or target-specific options with these defaults.
        var options = this.options({
            format: 'GeoJSON',
            // skipFailures: true,
            // projection: 'EPSG:4326',
            // options: null
            // timeout: 15000
        });

        // Iterate over all specified file groups.
        this.files.forEach(function(f) {
            var src = f.src.filter(function(filepath) {
                if (!grunt.file.exists(filepath)) {
                    grunt.log.warn('Source file "' + filepath + '" not found.');
                    return false;
                }
                else {
                    return true;
                }
            }).map(function(filepath, i) {
                grunt.file.mkdir(path.dirname(f.dest));

                var ogr = ogr2ogr(filepath);
                ogr.on('ogrinfo', console.error);
                ogr.on('error', console.error);
                ogr.format(options.format);
                var output = ogr.stream();
                output.pipe(fs.createWriteStream(f.dest));
            });
        });
    });

};

Adding some debug tracing shows that the source files exist and from the empty output I can see the right destination file is being created. But I'm now running into a brick wall of what to do next.

Thoughts and suggestions are welcomed!

Thanks

[wavded]

@vicchi unfortunately, I don't use nor know anything about how Grunt works. I am wondering if Grunt is somehow hijacking the STDIO streams so STDERR doesn't output to the console. What is this.files object look like in this context?

[vicchi]

Grunt should be just plain Node.js. The fact that my standalone code works fine, makes me suspect that this isn't a Grunt problem per se, but a synchronous vs. asynchronous problem.

I could test this via Grunt's async() support, but to do that, I'd need either a way of getting a final, task completed, callback back from ogr2ogr or a way of working out how to deal with the data that ogr2ogr.exec passes to its callback so I can get my final destination files written (and zipped if appropriate).

What I think is happening is that Grunt is assuming that ogr2ogr runs synchronously and isn't waiting for the (internal) callbacks that ogr2ogr uses to process the input and output files, so effectively Grunt finishes up whilst ogr2ogr is still doing "stuff". If that makes sense.

Any suggestions for how to synchronise with ogr2ogr actually completing?

Also, the contents of this.files are

[
  {
    "src": [],
    "dest": "data/dest/geojson/ne_110m_admin_0_sovereignty.geojson",
    "orig": {
      "src": [
        "data/src/ne_110m_admin_0_sovereignty.shp"
      ],
      "dest": "data/dest/geojson/ne_110m_admin_0_sovereignty.geojson"
    }
  }
]

... but I don't think this is a file specification problem.

[wavded]

I could test this via Grunt's async() support, but to do that, I'd need either a way of getting a final, task completed, callback back from ogr2ogr or a way of working out how to deal with the data that ogr2ogr.exec passes to its callback so I can get my final destination files written (and zipped if appropriate).

Streams returned from calling .stream() from ogr2ogr will emit a close event when all the processing is finished that you can listen for.

[vicchi]

I could test this via Grunt's async() support, but to do that, I'd need either a way of getting a final, task completed, callback back from ogr2ogr or a way of working out how to deal with the data that ogr2ogr.exec passes to its callback so I can get my final destination files written (and zipped if appropriate).

Streams returned from calling .stream() from ogr2ogr will emit a close event when all the processing is finished that you can listen for.

Thanks. I'll give that a try. I'm offline travelling for a few weeks now but I'll give it a go when I'm back and online.

[vicchi]

Progress. Had a spare few minutes before getting on a plane. Forcing Grunt to wait for the completion of ogr2ogr2 via the stream's close event works perfectly. If you're interested, the key is using this.async() and then calling the return value in the close event handler. See below.

'use strict';

module.exports = function(grunt) {
    var ogr2ogr = require('ogr2ogr');
    var fs = require('fs');
    var path = require('path');

    // Please see the Grunt documentation for more information regarding task
    // creation: http:https://gruntjs.com/creating-tasks

    grunt.registerMultiTask('ogr2ogr', 'ogr2ogr wrapper task for Grunt', function() {
        grunt.log.debug(JSON.stringify(this.files, null, 2));

        // Merge task-specific and/or target-specific options with these defaults.
        var options = this.options({
            format: 'GeoJSON',
            // skipFailures: true,
            // projection: 'EPSG:4326',
            // options: null
            // timeout: 15000
        });

        var self = this;
        // Iterate over all specified file groups.
        this.files.forEach(function(f) {
            var src = f.src.filter(function(filepath) {
                if (!grunt.file.exists(filepath)) {
                    grunt.log.warn('Source file "' + filepath + '" not found.');
                    return false;
                }
                else {
                    return true;
                }
            }).map(function(filepath, i) {
                grunt.file.mkdir(path.dirname(f.dest));

                var done = self.async();
                var stream = ogr2ogr(filepath).format(options.format).stream();

                stream.on('close', function() {
                    done();
                });
                stream.pipe(fs.createWriteStream(f.dest));
            });
        });
    });
};

I've loaded the output GeoJSON and Shapefile up in QGIS and all looks good. One final and unrelated question.

$ file ne_110m_admin_0_sovereignty.zip 
ne_110m_admin_0_sovereignty.zip: Zip archive data, at least v2.0 to extract
$ unzip ne_110m_admin_0_sovereignty.zip 
Archive:  ne_110m_admin_0_sovereignty.zip
  inflating: OGRGeoJSON.dbf          
  inflating: OGRGeoJSON.prj          
  inflating: OGRGeoJSON.shp          
  inflating: OGRGeoJSON.shx          

Is there any way to specify the name of the (compressed) shapefile (as opposed to the resultant ZIP archive) to be something other than OGRGeoJSON?

[DemersM]

@vicchi for your last question you can change the name of the output file by using the -nln (ie. new layer name) flag
for instance:

var shapefile = ogr2ogr('/path/to/spatial/file.geojson')
                .format('ESRI Shapefile')
                .options(['-nln', 'output_name'])
                .skipfailures()
                .stream()
shapefile.pipe(fs.createWriteStream('/shapefile.zip'))

[wavded]

Closing due to age, reopen if issue persists.