Edit files in place

[ghost]

With GNU Sed for example you can edit files in place:

-i[SUFFIX], --in-place[=SUFFIX]

       edit files in place (makes backup if SUFFIX supplied)

However I see no option for this with JQ, currently I must do this:

br=$(mktemp)
jq .charlie delta.json > "$br"
mv "$br" delta.json

Here is a workaround, Awk slurp:

jq .firstName alfa.json | awk 'BEGIN{RS="";getline<"-";print>ARGV[1]}' alfa.json

[karelv]

+1

[ipmcc]

+1

[nicowilliams]

Scripting around this is not an option? In shell:

tmpf=$(mktemp --tmpdir=$(dirname $f) -t)
jq ... "$f" > "$tmpf"
mv -f "$tmpf" "$f"

[nicowilliams]

I'm asking why it's not acceptable.

[ipmcc]

By analogy: Would it be acceptable if <insert your favorite word processor here> made you save to a different file every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is cumbersome. The fact that something as fundamentally stream oriented as sed has an in-place option is a pretty good indicator that this is common expectation/desire.

[nicowilliams]

I see. I've generally not bothered with sed's -i option. For me the
problem has been two-fold: a) it is (well, was) not portable enough, b) I
always have to check whether such a tool's in-place update functionality is
documented as working this way (rename into place), because I generally
need atomicity. And then there's Windows (where renames generally don't
work when there's open file handles for the file in question, unless all
the processes that opened it used an option to allow it).

On Mon, Jun 9, 2014 at 10:23 AM, ipmcc [email protected] wrote:

By analogy: Would it be acceptable if made you save to a different file
every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is
cumbersome. The fact that something as stream oriented as sed has an
in-place option is a pretty good indicator that this is common
expectation/desire.

—
Reply to this email directly or view it on GitHub
#105 (comment).

[nicowilliams]

Anyways, I can add this easily. My only concern is to limit the number of
new command-line options. Since this option couldn't be implemented in a
jq program, I'll probably do it. Though perhaps a generic utility to wrap
filters with might be better:

$ inplace some-file command args

which would run the given command with some-file as the standard input
and which would redirect the command's output to a temporary file in the
same directory then rename that into place. That's the Unix philosophy,
after all.

On Mon, Jun 9, 2014 at 10:58 AM, Nico Williams [email protected]
wrote:

I see. I've generally not bothered with sed's -i option. For me the
problem has been two-fold: a) it is (well, was) not portable enough, b) I
always have to check whether such a tool's in-place update functionality is
documented as working this way (rename into place), because I generally
need atomicity. And then there's Windows (where renames generally don't
work when there's open file handles for the file in question, unless all
the processes that opened it used an option to allow it).

On Mon, Jun 9, 2014 at 10:23 AM, ipmcc [email protected] wrote:

By analogy: Would it be acceptable if made you save to a different file
every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is
cumbersome. The fact that something as stream oriented as sed has an
in-place option is a pretty good indicator that this is common
expectation/desire.

—
Reply to this email directly or view it on GitHub
#105 (comment).

[nicowilliams]

Ah, there's a sponge(1) that almost works this way:

http:https://joeyh.name/code/moreutils/
http:https://backreference.org/2011/01/29/in-place-editing-of-files/

I think both, sponge(1) and my idea of an inplace(1) would be very useful
generally and very much in keeping with the Unix philosophy.

Would that work for you?

[nicowilliams]

@svnpenn @ipmcc @karelv

Regardless of how you feel about this issue, what think ye of this:

https://github.com/nicowilliams/inplace

?

[drorata]

Is the -i option available or not? In the linked commit it says that this feature was removed.

[pkoppstein]

@drorata - It was (as best I can tell) never available in any official jq release, and is certainly not available in jq 1.4 or 1.5. It was added as an experimental feature on Jul 20, 2014, and removed on Mar 5, 2015.

[pkoppstein]

@svnpenn - Although there are obviously circumstances when -i would be convenient, jq is more like awk than sed. The earlier attempt to endow jq with -i confirms that such attempts will be fraught with difficulties. awks do not have an -i option, and I think it is asking too much of the jq developers/maintainers to devise and implement such an option, especially given there are easy workarounds.

[nicowilliams]

Some of the refactoring to move I/O from main.c was done with in-place editing in mind, and is necessary to implement sed-style in-place editing. But it's not a high priority for me at this time, meaning: send us a PR if you want this done soon :)

[pkoppstein]

@svnpenn - My apologies if I over-generalized, but the GNU awk documentation characterizes "inplace editing" as "an extension" (https://www.gnu.org/software/gawk/manual/html_node/Extension-Sample-Inplace.html). Also, your example does not work with the GNU awk installed on my machine (4.0.2), and the awk that comes with OS X 10.9.5 does not support a "-i" option at all.

[dtolnay]

I agree that jq can and should do in-place. The earlier problems were due to a poor original implementation.

[nicowilliams]

The original implementation was poor and, in petticoat, had bad semantics. I refactored the way I/O is done in main.c in order to make it easier to later add sed-like `-i` -- it should be easy enough now. Send a PR or be patient :)

[danielbayley]

Is this feature in the next release? I found it in 'release candidate 1', or is that the version that was retracted?

[nicowilliams]

@danielbayley No, it got taken out. 1.5rc1 implemented the wrong -i semantics., sorry.

[nicowilliams]

There's no PR for it at the moment. Feel free to send one :)

[ghost]

+1

[BrendanThompson]

any action on this one?

[devopscast]

because -i works nicely for multiple files

meh.. that's what tee(1) is for and that only works if your iterating over files which is again anti-practice;
for x in foo fu bar; do something $x; done Is a lot slower than xargs -I{} -P3 something {} <<< foo fu fubar or if one passes many '-i filename(s)' flags. In both cases they make for really unreadable scripting and a lot of codegolf'd one offs. Which is why I stand on the grounds of anti-practice.

My debate really isn't about should jq have an inplace flag, that might benefit many. But more about good scripting in general.

Most commands should be seen equally to a library method. jq just happens to give us the missing jsonpath method which other interpreter languages have baked in. In most programming best practices the fewer arguments you give to the method/function the better. Anything longer should be an interpreted file for repeatability (see awk -f for an example). Need that method/function to do more, write a wrapper to it.

[gentunian]

@omgdenzuko so GNU sed -i flag to you is a bad practice?

[panwarab]

I am honestly amazed. 😲

[samliddicott]

It's the modern authoritarian way, justifying itself with the days when there was no choice. "Must stop people using features that I don't know how to use" And so missing features must be replaced by idiomatic expressions everywhere they are needed, and are even harder to get right. Those who don't know how to use -i don't need to use it, but they seemed so determined to prevent anybody else succeeding with it. Do they really look at people using sed -i and not know what is going on, or think that they would be just as easily able to parse an idiomatic loop and prove that it did the same thing? Surely these are the kind of people who write interview questions. 1. Prove that this expression is functionally the same as sed -i on the list of files. (Time allowed 10 minutes) 2. Explain why it is superior to sed -i (Time allowed 5 minutes) 3. Explain how it is more obvious that the expression replaces the files in place than it is that -i replaces the files in place (Time allowed 45 minutes) Bonus points of the interviewee can come up with their own expression which only fails on unlikely corner cases, or "rare" race conditions

…

On Mon, 9 Aug 2021, 22:27 Abhiroj Panwar, ***@***.***> wrote: I am honestly amazed. 😲 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABL7PTCLWIFX6ORL2KSJXQTT4BB4PANCNFSM4AEYTROA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[MadsBuchmann]

I would like this feature 😄

[loynoir]

Although I use below, still want '-i' or '-w'

jq 'del(.packageManager)' package.json | sponge package.json

[ipleten]

It can be done with:

echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' /etc/docker/daemon.json)" > /etc/docker/daemon.json

[devopscast]

Don't even need the echo.

…

On Wed, Oct 20, 2021, 18:02 Iaroslav Pleten ***@***.***> wrote: It can be done with: echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' /etc/docker/daemon.json)" > /etc/docker/daemon.json — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUILSHGDJM7AUS5KQZ34FTTUH5DAJANCNFSM4AEYTROA> .

[loynoir]

Don't even need the echo.
…
On Wed, Oct 20, 2021, 18:02 Iaroslav Pleten @.***> wrote: It can be done with: echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' /etc/docker/daemon.json)" > /etc/docker/daemon.json — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUILSHGDJM7AUS5KQZ34FTTUH5DAJANCNFSM4AEYTROA .

@omgdenzuko Yes, you need, unless you are not using bash/zsh/...

[devopscast]

Ok.. Since when did any posix shell not support proper stdout redirecting in the past 40-ought years? No. Echoing a subshell's output as a string is, I repeat on K&R's graves, not needed. Its like piping into cat from cat. jq '<expression>' > output.json works on all posix systems. Even then piping into tee output.json is acceptable. I should know its something I do a fing lot for work. But hey.. I'd love to see the asciimina of where I'm wrong.

…

On Wed, Oct 20, 2021, 18:54 loynoir ***@***.***> wrote: Don't even need the echo. … <#m_-1820777542152595716_> On Wed, Oct 20, 2021, 18:02 Iaroslav Pleten *@*.***> wrote: It can be done with: echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' /etc/docker/daemon.json)" > /etc/docker/daemon.json — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment) <#105 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUILSHGDJM7AUS5KQZ34FTTUH5DAJANCNFSM4AEYTROA . @omgdenzuko <https://github.com/omgdenzuko> Yes, you need, unless you are not using bash/zsh/... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUILSHENEPYRKJFKS5YGTK3UH5JDNANCNFSM4AEYTROA> .

[loynoir]

@omgdenzuko

Don't even need the echo.

$ echo '{"a":"b"}' > daemon.json && jq '. + {"registry-mirrors": ["https://docker-mirror"]}' daemon.json > daemon.json && cat daemon.json

$ echo '{"a":"b"}' > daemon.json && echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' daemon.json)" > daemon.json && cat daemon.json

{
  "a": "b",
  "registry-mirrors": [
    "https://docker-mirror"
  ]
}

[loynoir]

@omgdenzuko
I'm not saying I'm disagreed.

jq '<expression>' > output.json works on all posix systems.

But not agree with your

Don't even need the echo.

Because jq '<expression>' inplace.json > inplace.json truncate inplace,
while @ipleten using echo "$(jq '<expression>' inplace.json)" > inplace.json to edit files in place

[devopscast]

Yeah that's the usual *sh not holding a file buffer after a return from subshell. It's exactly why tee(1) exists. Just pipe stdout to tee or read so one gets stdout as a memory block to flush to disk. That's what your echo is doing while adding complexity, it's capturing stdout from the subshell then parsing it as a string, looking for shell expansions to evaluate then dumping it to its stdout where the shell then redirects it to a file. jq &1 pipe tee file eliminates the evaluation and just writes to the file.

…

On Wed, Oct 20, 2021, 22:33 loynoir ***@***.***> wrote: @omgdenzuko <https://github.com/omgdenzuko> Don't even need the echo. $ echo '{"a":"b"}' > daemon.json && jq '. + {"registry-mirrors": ["https://docker-mirror"]}' daemon.json > daemon.json && cat daemon.json $ echo '{"a":"b"}' > daemon.json && echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' daemon.json)" > daemon.json && cat daemon.json { "a": "b", "registry-mirrors": [ "https://docker-mirror" ] } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUILSHHMYVGEGUUUQVFNLD3UH6CYLANCNFSM4AEYTROA> .

[loynoir]

@omgdenzuko
Oh, seems I misunderstood your commment

Don't even need the echo.

On ipleten's approach

echo "$(jq '<expression>' inplace.json)" > inplace.json

To

jq '<expression>' inplace.json > inplace.json

But seems you actaully means

jq '<expression>' inplace.json | tee inplace.json

I prefer sponge or echo approach then tee approach

https://stackoverflow.com/questions/33638511/differences-between-sponge-and-tee

[pkoppstein]

Using > or tee is at best a bad idea unless your shell or version of tee specifically documents that it can be used in the desired way.

For more on this, see http:https://askubuntu.com/questions/752174

Using sponge is often the way to go.

[ipleten]

Hey!
I'm glad I triggered such discussion and really glad so many people know *nix shell. sponge isn't included by default in almost all docker containers(frankly I've never heard about it before).
I just put one of many possible solutions.
Thanks!

[devopscast]

@ipleten in any debian based distro sponge is apart of the moreutils and its not part of POSIX or heirloom *nix so that's why its not include by default.

A list of supported package managers to install sponge and it copy-pasta command can be found at https://command-not-found.com/sponge

[samliddicott]

This repeated discussion, the usual mistakes, the file races, repeated all over the world (not only here) is why in-place option is needed and why it is futile to argue about how the unix philosophy renders it not strictly necessary. It is needed, because too often people can't get the workarounds right. The sheer number of hours wasted must be incredible and only get worse as jq becomes more popular in spite of this obvious deficiency. Sam

…

On Thu, 21 Oct 2021, 20:42 Dwight Spencer, ***@***.***> wrote: @ipleten <https://github.com/ipleten> in any debian based distro sponge is apart of the moreutils and its not part of POSIX or heirloom *nix so that's why its not include by default. https://command-not-found.com/sponge — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABL7PTBNP4CXZ3O26VAQK7DUIBUJVANCNFSM4AEYTROA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ryanotella]

yq does support inplace editing and can produce normal JSON. CLI API is different however.

$ echo '["first"]' > message.json
$ yq -i -ojson eval '. += ["second"]' message.json 
$ cat message.json
[
  "first",
  "second"
]

https://mikefarah.gitbook.io/yq/

[shakibamoshiri]

I use yq,
For advanced users this -i (in-place update) is needed, hope be added to jq

yq -iP '.Email.Port=3030' config.json -o json

• -i in place update
• -P pretty print
• -o output should be json

[samliddicott]

After nearly 10 years of jq maintainers arguing against in-place editing on the grounds that there are difficult and obscure ways around that delightful limitation, we now also have jaq, written in rust, with in-place editing of multiple files, and fixing various jq bugs:

https://github.com/01mf02/jaq#readme

[drm]

I hope this PR can once and for all end this debate. It's a really simple change that follows exactly the solution that was explained in the Kernighan and Pink document as mentioned earlier, plus it doesn't impose any big changes on how the internals work.

Re yq and jaq: https://xkcd.com/927 comes to mind :P

[sdavids]

While we are waiting ...

Minify all *.json files in a given directory:

find . \
  -type f \
  -name '*.json' \
  -exec sh -c 'mv "$1" "$1.tmp"; jq -c . "$1.tmp" > "$1"; rm "$1.tmp"' shell {} \;

[luckman212]

@sdavids what does the shell right before the {} do in the above statement?

[sdavids]

The string sh. This will be available as $0 inside the script, and if the internal shell outputs an error message, it will prefix it with this string.

https://unix.stackexchange.com/questions/389705/understanding-the-exec-option-of-find#answer-389706

I used shell instead of sh so one does not mistakenly think that one can select a specific shell in that position—but the label is up to taste and whether you use $0 or not.

[gentunian]

github community is getting weird...