Remove scripts created by batch

[PaulWessel]

See the forum for context. Without -W the content of the prefix script would leave behind script files produced by batch. These are now removed via this PR. Also changed the -Zs to just -Z in the documentation example since there is no s directive.

[Esteban82]

It partially worked. All the files were removed but I still have the folder.

[PaulWessel]

Yes, but it is possible the scripts will produce things in there. I need to look at the logic a bit more to see if it is always "safe" to remove the prefix directory or not.

[PaulWessel]

I am concerned by these lines in the docs under -W:

The product files will still be placed in the prefix directory. The workdir is removed unless -Q is specified for debugging.

I am not sure if that is actually true (for starters) or desired.

[PaulWessel]

We also say (under Parameters):
Note: Any product(s) made by the processing scripts should be named using **BATCH_NAME** as their name prefix as these will be automatically moved up to the starting directory upon completion.

So there are many mixed messages here which I think gives us some flexibility in deciding how to solve it. Much of the machinery of batch was inherited from movie, but there are some big differences between movie and batch (and some similarities):

• movie most likely will produce a final movie (Mp4 or gif) and it is moved up and placed at the same level as the prefix directory. However, if you choose to save the individual PNG files then they are all in the prefix directory.
• batch may or may not make a final product; it could be none or many, and it such cases it seems very messy to have all these files moved up to the same level as the prefix directory (what is the point of that directory if not to contain all these products?). So, after a failed batch run the user will have to delete a bunch of files all over the top directory instead of just the prefix directory.

I think we should declare that we have a bug here and that all products named via BATCH_NAME shall be present in the prefix directory when the command finishes. There is no way of knowing if there is a single final master product (such as the countries.pdf book based on all the individual files in the man page example) or a bunch (such as all the grids produced in the other example). Furthermore, the user could always place a mv command in the post-script to place result(s) elsewhere.

Finally, since the pre.sh script may create some temporary files in the prefix folder it is not possible to know in general what that is (and delete them specifically), but one solution is to get the list of all the files present in prefix after the run is completed and then delete all those does not match the BATCH_NAME prefix.

So in summary, I propose this (modified) behavior and corresponding documentation improvements:

1. Let the prefix directory contain all products that were named using BATCH_NAME in the main script.
2. Remove all other temporary files that were created in the prefix directory
3. If the prefix directory is empty after these actions are taken then we delete the prefix directory.

Comments on these, @GenericMappingTools/core ?

[PaulWessel]

I am making one change to the above suggestions: While products called BATCH_TAG.* will be left in the prefix directory (the equivalent of the PNG files like prefix_0001., prefix_0002.), anything called BATCH_NAME.* will be moved up to the top directory. This gives nice flexibility of leaving lots of numbered files in the prefix directory while letting final summaries (such as countries.pdf) be placed outside (and thus removing the empty prefix directory).

[PaulWessel]

I have now implemented the items discussed above. Perhaps @Esteban82 would like to test? The key changes are

1. Files named BATCH_PREFIX.* are moved to the starting directory
2. Files named BATCH_NAME.* are left in the prefix (or workdir) directory
3. All other temporary files are removed
4. An empty prefix or workdir directory will be removed upon completion

The two batch examples have been modified slightly. Because some things might be OS-dependent it would be good to test this on Windows and Linux as well as macOS.

[Esteban82]

It worked fine with the example 2 of batch.

Then I try it in another script where I use this command in the post script:

awk -F'\t' '{printf "mv GRACE_%1.1d.png \"%s.png\"\n", NR-1, \$1}' ../Estaciones.txt | sh -s

The files are rename but I all the scripts are not deleted (main.sh, include.sh, post.sh in the main directory and batch_init.sh, batch_job.sh and batch_postfligth.sh in the workdir). Not sure if there is an issue with the previous command or with batch module.

In the terminal I got this error:

batch [ERROR]: Running postflight script batch_postflight.sh returned error 256 - exiting.
free(): double free detected in tcache 2

[PaulWessel]

Hard to tell what is happening but if the postflight script fails then we exit before we examine the files and delete the ones you found are there. Could you place your full post script here please? You could also run with -Qs with just a few jobs so you can run that script manually and see if there is something odd with it.

[Esteban82]

I am not sure if the postflight fails because all the files were renamed fine.

The full script is

#!/usr/bin/env bash
title=GRACE
cat << EOF > include.sh
	PROJ=-JX7c
	GEO="Datos/Datos_estaciones.txt"
	Archivo=${title}_Estadisticas.txt
EOF
echo \# Long Lat slope y0 N R Corr Estacion > Final.txt
cat << EOF > Estaciones.txt
EST1
EST2
EOF
cat << EOF > main.sh
gmt begin \${BATCH_NAME} png
	gmt convert Datos/ALT_GRACE_\${BATCH_WORD0}.txt -e~9999 > temp
	REGION=\$(gmt info temp -i2,1 -I20/2)
	gmt basemap \$REGION \$PROJ -B+n
	gmt basemap -B+t"\${BATCH_WORD0}"
	gmt basemap -Bxaf+l"TWS (mm)" -BSn
	gmt basemap -Byafg+l"HH (mm)" -BWe
	gmt makecpt -Crainbow -T1/12/1 -F+c1 -H > a.cpt
	gmt convert temp -i2,1,0 --FORMAT_DATE_OUT=mm --FORMAT_CLOCK_OUT=-  | gmt plot -Ss0.15c -Ca.cpt
	gmt regress temp -i2,1 -Fxm $1 $2 | gmt plot -Wthin
	geo=\$(gmt convert Datos/Datos_estaciones.txt -e\${BATCH_WORD0} -o0,1  --FORMAT_FLOAT_OUT=%.7f)
	regr=\$(gmt regress temp -i2,1 -Fp -o5,6,0,10,9  --FORMAT_FLOAT_OUT=%.3g)
	echo \$geo \$regr \${BATCH_WORD0} >> ../../Final.txt
gmt end
EOF
cat << EOF > post.sh
awk -F'\t' '{printf "mv GRACE_%1.1d.png \"%s.png\"\n", NR-1, \$1}' ../Estaciones.txt | sh -s
EOF
gmt batch main.sh -TEstaciones.txt+w"\t" -Vi -N$title -Iinclude.sh -Z  -Sfpost.sh

Data for the script: Datos.zip

[PaulWessel]

It ran for me after I removed the ../ in fromt of Estaciones.txt. Probably related to the updated batch.c where post now runs inside the working directory.

[Esteban82]

It ran for me after I removed the ../ in fromt of Estaciones.txt. Probably related to the updated batch.c where post now runs inside the working directory.

Sorry. I don't understand if you have any problem or not (besides the ../).

[PaulWessel]

Once I removed the ../ I had no problems - it ran fine. WIth the ../ it failed.

[Esteban82]

Once I removed the ../ I had no problems - it ran fine. WIth the ../ it failed.

Mmm. I am getting an error in both cases.
Without "../" = awk: fatal: no se puede abrir el archivo «Estaciones.txt» para lectura (No existe el archivo o el directorio)

[PaulWessel]

Are you using the batch.c version from this PR? One thing that this update does is to change the current directory when the post script is run. It used to run from outside the -N work directory but now, like in movie, it is inside so that it can benefit from knowing where all the data files are. While gmt modules can find data files both in the top directory and in the work directory (since we set GMT_DATA path in the scripts that are created, there is no such mechanism for arbitrary unix commands of course (e.g., gawk) so here you need to be exact. I suspect you are running the above script with master or 6.1.1?

[Esteban82]

I am using Version 6.2.0_5e69197-dirty_2021.05.14 and the batch-delete branch.

[PaulWessel]

Current version seems to work great for me, with moving products up to top and leaving any name_##### files in the workdir which is then not removed. However, it causes a problem with the 2nd example on the man page in that the gmt end show command in the post script cannot find the plot since it has been moved up to the top. If the rule is that products shall move up then we cannot name this plot filter.pdf

[Esteban82]

For me works fine too if I removed the postflight script with the gawk command. With it, I still can't managed to renamed the files but most likely it is because I am using gawk wrong.

[PaulWessel]

Hi @Esteban82, I just updated this branch with master and fixed the conflicts (it has been sitting here sin 2021). Could you run your case again and see how it goes?

[Esteban82]

@PaulWessel I think that there is an error in the code.

I got this error while trying to build this branch:

[ 14%] Building C object src/CMakeFiles/gmtlib.dir/batch.c.o
/home/thor/Github/GenericMappingTools/gmt/src/batch.c: In function ‘GMT_batch’:
/home/thor/Github/GenericMappingTools/gmt/src/batch.c:998:9: error: ‘P_len’ undeclared (first use in this function)
  998 |         P_len = strlen (Ctrl->N.prefix);        /* Length of the prefix */
      |         ^~~~~
/home/thor/Github/GenericMappingTools/gmt/src/batch.c:998:9: note: each undeclared identifier is reported only once for each function it appears in
/home/thor/Github/GenericMappingTools/gmt/src/batch.c:1045:30: warning: implicit declaration of function ‘gmt_get_dir_list’; did you mean ‘gmt_get_dim_unit’? [-Wimplicit-function-declaration]
 1045 |                 work_files = gmt_get_dir_list (GMT, workdir, NULL);     /* Get list of all files in workdir */
      |                              ^~~~~~~~~~~~~~~~
      |                              gmt_get_dim_unit
/home/thor/Github/GenericMappingTools/gmt/src/batch.c:1045:28: warning: assignment to ‘char **’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
 1045 |                 work_files = gmt_get_dir_list (GMT, workdir, NULL);     /* Get list of all files in workdir */
      |                            ^
/home/thor/Github/GenericMappingTools/gmt/src/batch.c:1058:17: warning: implicit declaration of function ‘gmt_free_dir_list’; did you mean ‘gmt_free_list’? [-Wimplicit-function-declaration]
 1058 |                 gmt_free_dir_list (GMT, &work_files);
      |                 ^~~~~~~~~~~~~~~~~
      |                 gmt_free_list
/home/thor/Github/GenericMappingTools/gmt/src/batch.c:1064:28: warning: assignment to ‘char **’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
 1064 |                 work_files = gmt_get_dir_list (GMT, workdir, NULL);     /* Get list of all files in workdir */
      |                            ^
gmake[2]: *** [src/CMakeFiles/gmtlib.dir/build.make:856: src/CMakeFiles/gmtlib.dir/batch.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:735: src/CMakeFiles/gmtlib.dir/all] Error 2
gmake: *** [Makefile:156: all] Error 2

[PaulWessel]

Sorry, what happens when merging Nov 2023 master into 2001 branch (> 1000 commits and conflicts). I just did not notice that the build failed before I added it. Now fixed I hope.

[Esteban82]

It is working fine.

[Esteban82]

I think we can merge it.