Thread-safe setting of variables

[hillu]

If I read libyara/rules.c correctly, when there are multiple threads
sharing a ruleset for scanning, there is no way to safely define
external variables.

I use variable definition to pass trivial information such as file or
process names that are used within some of my rules. Of course, I
could use one copy of the ruleset per thread, but in my use case that
would mean a much larger memory footprint. (Also, this would defeat
the purpose of YARA's support for multiple threads.)

The yr_rules_define_*_variable functions operate on a global
structure hanging off the YR_RULES struct without any locking. Even
though yr_rules_scan_mem_blocks eventually copies everything into
context.objects_table before starting the actual scan, I see no way
to ensure that the variables hanging off rules->externals_list_head
are concsistent -- even if I protect the individual
yr_rules_define_*_variable calls with a mutex.

Am I missing something?

[plusvic]

You are right, yr_rules_define_*_variable aren't thread-safe. You can't assign a thread-specific value to a given variable, all threads share the same value which should be assigned by a single thread and shared by all threads calling yr_rules_scan_*.

As you said, the most important limitation here is that you can't pass values that depends on the file being scanned, like filename, or whatever. This could be solved by providing a separate structure to each thread to store variable values, but that would break existing code that assumes that once the main thread assigns value to a variable all other threads inherit that value.

[bmagistro]

I have hit this issue as well and will likely just be making multiple copies of the rules. Is there any chance the yr_rules_scan_* functions can be extended to include an (optional) variable list? Under the hood it may be possible to keep the calls the same and point this new argument to the rules/compiler structure's external variable list when it isn't present. The thinking being this might avoid breaking existing code while allowing an additional path to be added where variables could be defined per scan call. Yes this means that all variables would need to be defined at all times even if not in use but I see this as a issue for the caller and not an issue for yara to solve.

Thoughts? If this sounds like a possible approach I'd be willing to try and code some of this up but not sure how successful I'd be.

[hillu]

@bmagistro What you describe is pretty much the path I had in mind.

[bmagistro]

@hillu I think I have something, I'm just now starting to test things using the new functions and providing the vars to the scan function.

Edit: It doesn't work (yet), it seems to work fine when not trying to pass an ext to the scan function, passing it to the scan function doesn't seem to match/apply. This is going to take a bit longer, it isn't as straight forward as I thought.