Parser question: expressions vs statements

[orefalo]

Hi again...

I am a bit puzzled as to how the parser works. I did read the chapter, yet I don't understand why the bellow code doesn't work

murex-utils » 44 * 4096 / 1073741824
0.0001678466796875
murex-utils » (44 * 4096) / 1073741824
0.0001678466796875
murex-utils » ((44 * 4096) / 1073741824)
((44 * 4096) / 1073741824)
murex-utils » expr ((44 * 4096) / 1073741824)
0.0001678466796875

murex-utils » round ((44 * 4096) / 1073741824) 0.99
Error in `round` (0,1): cannot convert '(44 * 4096) / 1073741824' to a floating point number: strconv.ParseFloat: parsing "(44 * 4096) / 1073741824": invalid syntax

murex-utils » round (44 * 4096) / 1073741824 0.99
Error in `round` (0,1): invalid parameters. Expecting `round <value> <precision>

murex-utils » round (expr ((44 * 4096) / 1073741824)) 0.99
Error in `round` (0,1): cannot convert 'expr ((44 * 4096) / 1073741824)' to a floating point number: strconv.ParseFloat: parsing "expr ((44 * 4096) / 1073741824)": invalid syntax

what am I doing wrong?

[orefalo]

The last two should work with no issues.

I should get the following post parsing,

stack 2: 0.99
stack 1: (44 * 4096) / 1073741824
stack 0: round

... I am not understanding how the expression evaluator works

[lmorg]

At the moment there isn’t any way to mix statements and expressions. This is where you’d need to invoke a subshell with ${}.

We did discuss before about the possibility of using parentheses to embed expressions inside statements but it’s not something I’ve coded yet. The main reason being that I wanted to let the idea brew a little longer in case I came up with any other elegant solutions (once this would be implemented, we’d likely then be stuck with it. So I wanted to make sure the right design was put in place).

[orefalo]

Right on spot then! Because now I do understand what we were discussing before. ;-)

Still... I don't understand how your command line evaluator works... (I will go hunt for the code)
my previous experience in that territory tells me that the input should be parsed into a stack, with metadata for each entry

ie. round ((44 * 4096) / 1073741824) 0.99

stack 2: Number(0.99)
stack 1: Expr((44 * 4096) / 1073741824)
stack 0: Builtin("round")

from there, the command evaluator, will apply priorities and evaluations based on the metadata similar to a fn(fn(fn()))

Mind sharing how the high level internals work?

[lmorg]

There’s a couple of different parsers (albeit they share code so it appears like a single parser).

First a command gets parsed as an expression. If it is a valid expression then the parser moves on and tokenises the next command in the pipeline. However it is an invalid expression (eg variable <> 5 would be invalid because <> isn’t a valid operator) then it gets parsed as a statement (ie command parameter1 parameter2 …)

A valid statement is anything that follows the basic format of: value operator value where value could be a scalar ($ or ~ style variable or subshell), string / numeric value, or further expressions encapsulated in parentheses.

This is why error messages say (to paraphrase) “not a valid expression nor statement”.

During this early phase of parsing, expressions and statements are only lazily parsed. Eg parentheses are not calculated in expressions, values are not type checked, etc. The purpose is only to identify if the command is an expression or statement, and if the latter, to then tokenise the parameters. This means things like parentheses and subshells could hide bugs that are not found out until the interpreter is ready to execute that line of code (I have thought about optimising this process but given that the nature of shells being a meta programming environment, and the relative performance of Murex, it hasn’t really been a pressing concern).

so it’s a bit of a hack but it’s a hack that works surprisingly well.

The reason the hack exists is simply due to the fact that shells are littered with barewords. All strings are allowed to be barewords. I could enforce strings are quoted (like they are in expressions) but that would get really annoying really fast when the realisation of having to quote flags hits. Eg ”find” “.” “-name” “file name” would be annoying compared with the more common way of writing that: find . -name “file name”.

So since statements can have unquoted strings, it’s means there isn’t any strict grammar rules to which you can figure out what an expression should like like, should the parser be written to treat statements as valid expressions.

this is why Murex has this weird split personality when it comes to expressions and statements. expressions are incredibly useful in programming because most other shells force expressions to be a quoted string passed to a command, which is ugly and error prone. But expressions add a level of verbosity people aren’t used to when calling executables (worth remembering that both Windows and POSIX only support strings as parameters so anything Murex does with types is already a hack on top of that).

last thing worth noting is that expressions do actually get converted into a statement, expr. Having “naked” expressions in the command line is really just syntactic sugar for expr “expression”. Thus the following two command lines are actually the same:

• 5 * (5 + 3)
• expr “5 * (5 + 3)”

but I think as much confusion as the double parsing brings, it’s outweighed by the convenience of writing expressions as expressions rather than strings.

[lmorg]

Sorry, I didnt mean to close this.

[orefalo]

This is the type of discussion which would typically be great around a board and a live interaction. ;-)

Thank you for the details,
Fully agreed - expressions without quotes are better, simpler, more intuitive.

Yet I am still missing something, why isn't everything an expression?
Why would $variable <> 5 be parsed as "$variable <> 5" rather than

stack 2: number "5"
stack 1: string "<>"
stack 0: variable("variable")

The stack above will still fail evaluation, but you get my point...
What are the use cases to support two parsing "methods"?

In a perfect world, everything should land on a stack.
Am I missing something?

[lmorg]

Because command lines aren’t expressions. Eg

find . -name example isn’t a valid expression. To turn that into an expression you then need to wrap it in some kind of syntax to denote where the function starts and ends. Eg

find(. -name example)

Then it’s pretty easy to tokenise a commands parameters.

The issue is complicated even more in that command lines use space as a separator but expressions aren’t whitespace sensitive. So should 5 - 5 be parsed as

• 5, “-“, 5 or
• 5, -5 or
• “5-“, 5 or
• “5-5” or
• 0 ?

The only way to factor that into your parser is to either

1. have two passes: once as an expression and again as a statement.
2. Abandon the old way of writing command lines and instead have something a lot more like find(“.”, “-name”, “example”)

The very early prototype of murex (then called JSH iirc) experimented with option 2 but it was horrible to use.

[orefalo]

not into a valid expression, I agree... but it could be a valid stack...

It could parse find . -name example into the following stack +metadata

stack[3]: string("example")
stack[2]: string("-name")
stack[1]: string(".")
stack[0]: builtin("find")

for an expression like 5 - 5,
the output could be

stack 2: number(5)
stack 1: operator - 
stack 0: number(5)

There is obviously a difference between expressions and commands, in which a command starts with the action, while expressions don't. so maybe the parser could detect it and output a reversed polish stack, like this

stack 2: number(5)
stack 1: number(5)
stack 0: operator - 

from there.. stack 0 has the signature : function(stack): stack
which is relatively easy to evaluate and compose.

[lmorg]

I’m not really sure that solves the problem though. You still need to define function and expression boundaries. Eg in LISP it might look like:

(round (* 5 5) 1)

The equivalent in murex would look like:

round ${5 * 5} 1

The question is whether that’s verbose. Eg would this be more elegant:

round (5 * 5) 1

which is actually not too difficult to implement because parenthesis are already a special case token. I’ve just been holding out doing so because there are a few design decisions I’ve made in a rush that I would love to undo if it weren’t for breaking backward compatibility.

[orefalo]

+1 round (5 * 5) 1

[orefalo]

fine -
let's put this discussion on hold for now.

[lmorg]

I'm going to re-open this but turn it into a discussion because it's something that should be kept in our consciousness rather than closed and forgotten...at least until a satisfactory resolution has been decided.

[lmorg]

I’m think about maybe having parentheses as a shortcut for inlining an expression. And function-name(parameters…) as a shortcut for inlining a single function.

this would also unify expressions and statements a little better because a statement then is function(parameters…) but the parentheses are optional under some circumstances

[lmorg]

Started work on this idea: #677 (note I'm working off a different branch for POC)

[lmorg]

dev » a=123
dev » b=45
dev » c=round(($a/$b) 0.0)
dev » $c
2.7

In the above example, c=round(($a/$b) 0.0) is an expression which:

• round(...) is a command (in this instance a builtin but it could be any command, function or alias)
• ($a/$b) is an expression inlined into a statement. Could be the same as `» round: ($a/$b) 0.0

#677 (comment)

[orefalo]

looks really neat and intuitive

[lmorg]

Now merged into develop

function(parameters...) now work in statements as well. Biggest thing that might catch people out is:

» echo uname(-o)foobar

outputs:

Darwin foobar

because anything that directly follows function(parameters...) is a new token. so uname(-o)foobar is parsed as:

command:     echo
parameter 1: output of `uname(-o)`
parameter 2: string "foobar"

even though there isn't any whitespace between ) and f in uname(-o)foobar

---

This behaviour differs from a subshell where whitespace is important. eg

» echo ${uname -o}foobar
Darwinfoobar

because it is parsed as

command:     echo
parameter 1: output of subshell (`uname(-o)`) + string ("foobar")

---

I'm undecided about which way is the "correct" way.

1. having ) terminate a parameter in C-style functions would make them behave the same in expressions as well as statements (function(parameters...) is a resolvable value, like (5*5) would be.
2. but also having ) require whitespace afterwards to start a new parameter in statements would make it consistent with subshells (${...}) but at the expense of being inconsistent between expressions and statements.

This is just a small detail though -- at least compared to the significance of the inclusion of function(parameters...) style syntax

[orefalo]

love it

[orefalo]

echo uname(-o)foobar

shouldn't the output be Darwinfoobar ?

[orefalo]

Agree with your last statement, it's small detail. leave it as is for now and let see if you get feedback.
I will resume conversion of my personal scripts -

[orefalo]

and I am done with my rewrite of the linux free command for osx

# function: free (description: "an osx substitution for linux 'free' command")
function: free {
	if { os darwin } then {
		# Use vm_stat and sysctl to get system info
		vmstat = @{vm_stat | regexp 'f/([0-9]+)/'}

		#out $vmstat

		# "Pages free:" 
		pfree=$vmstat[1]
		# "Pages wired down:"
		pwired=$vmstat[6]
		# "Pages inactive:" 
		pinact = $vmstat[3]
		# "Anonymous pages:"
		panon = $vmstat[14]
		# "Pages occupied by compressor:"
		pcomp = $vmstat[16]
		# "Pages purgeable:" 
		ppurge = $vmstat[7]
		# "File-backed pages:"
		pfback = $vmstat[13]

		total_mem = ${sysctl -n hw.memsize}

		#out $pfree $pwired $pinact $panon $pcomp $ppurge $pfback $total_mem

		# Arithmetics
		total_mem = $total_mem / 1073741824
		round ($pfree * 4096 / 1073741824) 0.99 -> set pfree
		round ($pwired * 4096 / 1073741824) 0.99 -> set pwired
		round ($pinact * 4096 / 1073741824) 0.99 -> set pinact
		round ($panon * 4096 / 1073741824) 0.99 -> set panon
		round ($pcomp * 4096 / 1073741824) 0.99 -> set pcomp
		round ($ppurge * 4096 / 1073741824) 0.99 -> set ppurge
		round ($pfback * 4096 / 1073741824) 0.99 -> set pfback

		# OSX Activity monitor formulas
		# thanks to http:https://apple.stackexchange.com/questions/81581/why-does-free-active-inactive-speculative-wired-not-equal-total-ram
		round ($pfree + $pinact) 0.99 -> set free
		round ($pfback + $ppurge) 0.99 -> set cached 
		round ($panon - $ppurge) 0.99 -> set appmem
		round ($appmem + $pwired + $pcomp) 0.99 -> set used

		# Display the hud     
		out '              total     used     free   appmem    wired   compressed'
		out "Mem:           $(total_mem)Gb   $(used)Gb   $(free)Gb   $(panon)Gb   $(pwired)Gb   $(pcomp)Gb"
		out "+/- Cache:            $(cached)Gb   $(pinact)Gb"

		swapusage = @{sysctl -n -o vm.swapusage | regexp 'f/([0-9]+.[0-9]+)/'}

		if { $swapusage[0] != "0.00" } then {
			round ($swapusage[1]*100/$swapusage[0]) 1 -> set swap_percent
			out "Swap($(swap_percent)%):   $swapusage[0]Mb   $swapusage[1]Mb   $swapusage[2]Mb"
		}	

		load = @{sysctl -n -o vm.loadavg  | regexp 'f/([0-9]+.[0-9]+)/'}
		out "Load Avg:      $load[0] $load[1] $load[2]"
	} else {
		command free $argv
	}
}

[lmorg]

Thanks for the feedback. A few questions if you don't mind:

I find the usage of = are -> set for assignments inconsistent. I am starting to get a grip on it, but it's not trivial for beginners

Are you able to elaborate on this please?

I was surprise to see that $swapusage[0] is not typed as a float/number, I have to write it as $swapusage[0] != "0.00"

That is because you are defining $swapusage from vmstat (which is textual output):

swapusage = @{sysctl -n -o vm.swapusage | regexp 'f/([0-9]+.[0-9]+)/'}

What should happen, though, is that types are auto-converted when doing these comparisons if strict-types is disabled in config (which it is by default). This works for integers:

» "0" == 0
true

but it looks like it's not working for floating point numbers:

» "0.00" == 0
false

so that is a bug I need to fix

[lmorg]

As an aside, it might make more sense to have your code read:

if { os darwin } then {
    function free {
        ...
    }
}

that way free is only defined as a function if os darwin is true and thus you don't then need the following else block because free wouldn't be defined on non-darwin operating systems (ie you'd automatically fallback to the hosts own implementation).

} else {
    command free $argv
}

also $argv is incorrect. You need @PARAMS: https://dev.murex.rocks/user-guide/reserved-vars.html#params-json

[lmorg]

I've pushed a fix for floating point type conversations with == and != operators to develop. eg

» "0.00" == 0
true

Also worth noting that there are "like" (~~) and "not like" (!!) operators too:

» "0.00" ~~ 0
true

I need to document these but will give you a quick explanation here:

• they first check if it looks like a number (eg an int, float or string with numeric values) and if so, normalises it (eg 0.00 will be converted to 0, 0123 to 123, etc)
• then it is converted to a string (eg false becomes "false", 123 becomes "123", etc)
• then leading and trailing whitespace removed
• then lowercased
• and then compared

For example:

» "   TRUE   " ~~ true
true
» "0001.23000" ~~ 1.23
true

The intention for ~~ and !! is these kinds of scenarios where you're using regexp and other hackiness to extract data from (for example) STDOUT and you want a way of checking if values are equivalent without having to think about type casting, cleaning the formatting up, etc.

Most of the time == and != should be enough but sometimes you want something with a little more 'magic' when dealing with the ugliness of unformatted command line outputs.

[orefalo]

very nice, i like the ~~ operator, it will do in my case

[orefalo]

and yes... it's a nice feature that is worth documentating ;-)