Could admiral have a "verbose" mode where useful info is printed to the log after each function run?

[manciniedoardo]

One possible example

After running something like:

derive_param_computed(
  advs,
  by_vars = exprs(USUBJID, VISIT),
  parameters = c("SYSBP", "DIABP"),
  set_values_to = exprs(
    AVAL = (AVAL.SYSBP + 2 * AVAL.DIABP) / 3,
    PARAMCD = "MAP",
    PARAM = "Mean Arterial Pressure (mmHg)",
    AVALU = "mmHg"
  )

We could print out:
<dataset_name> has xxx rows and yyy columns

or even more detailed:
<dataset_name> has xxx new rows and yyy new columns

We could even include more info about new PARAMCDs that have been derived.

Thoughts/Questions/Comments

• Such an effort would synergise quite well with ongoing discussions around creating a SAS-style log for admiral functions
• We could control whether this is turned on or off in an admiral option
• such functionality would probably live in admiraldev, kind of like our testing functions
• how would this work when piping multiple calls in sequence?
• We'd need a joint effort across extension packages as well to standardise (if implemented)

@pharmaverse/admiral Thoughts? Is it too much of a blue sky thought? @pharmaverse/admiral_comm would this be a useful addition

[bms63]

That seems like a lot of work!! Are folks asking for SAS-like log at Roche?

Would a print method be better?

...or could we just show how to use message() and warning() to output things like this to the log if folks want this level of detail?

[bms63]

To me reworking functions to include verbosity arguments has potential to introduce more problems (performance, bugs, etc) to admiral.

Obviously I can be persuaded if folks see this as more valuable use of resources - just seems like we are trying to make R behave like SAS, which always rings alarm bells.

[manciniedoardo]

I see your point @bms63, agree that we'd have to be careful. Part of my reasoning behind suggesting it was that roak has similar behaviour and it's useful to check whether things are working as expected at a glance.

[bms63]

Interesting - I guess a demo of the roak functionality would be in order. Could be nice at next admiral meeting.

[manciniedoardo]

let me see what i can organise! Might not be for the next one but the one after.

[bundfussr]

The needs for roak may be different because in roak a single function call creates the complete SDTM dataset.

[bundfussr]

I see two use cases where users could benefit from more information:

• Batch runs: If a batch run fails or produces different results than the interactive run, it is really hard to find the reason the batch run provides just the beginning of the function calls and errors and warnings.
• Complex functions which perform several steps like derive_extreme_event() or derive_vars_joined(). If the result is not as expected, it is hard to investigate. As an admiral developer I can use debug mode to investigate intermediate results. But for users it is difficult because they need to understand the function code first.

Some thoughts/questions:

• Which functions should write to the log? Just derive_*() functions or also computations, higher order functions, or the filter_*() functions?
• How do we handle nested calls, e.g., derive_vars_computed() calls derive_param_computed(). If a users calls derive_vars_computed() and derive_param_computed() issues a message that x records were added, it could be confusing.
• Is a text file the best option for a log? Why not storing intermediate datasets or other information like function calls in an environment and write it to the disc?
• We should take enough time to discuss this topic thoroughly. The implementation will require a lot code changes. Thus we should be sure that it helps the users.

[StefanThoma]

I agree that we should not just do this for the sake of it, and at the moment I am not 100% sure of the added benefit.
I do however find the log that tidylog provides quite useful:
https://www.r-bloggers.com/2020/01/tidylog-logging-your-pipelines/

[bundfussr]

I think using tidylog would require a lot of work because in ADaM scripts most dplyr/tidyr functions are not called directly. Turning on tidylog in admiral would generate many messages but without providing the corresponding dplyr calls. Most likely this wouldn't be helpful. To make it useful we would need to decide for each dplyr call if a log message should be generated and in addition we would need to provide additional information like which dataset was filtered dataset or dataset_add.

[ddsjoberg]

I wouldn't suggest adding SAS-like logging in admiral. It will be A LOT of work to add it and even more to maintain. I get the same alarm bells as @bms63

When we're in running in batch and things go wrong, I would hope the error messaging is clear what went wrong. In cases when it's not clear, let's improve that messaging.

Of course checking our results after derivations is incredibly important. Rather than relying on a verbose SAS-like log, perhaps we can brainstorm other avenues that accomplish a similar goal of of transparency and results checking.

We could perhaps instead pilot a few functions outside of admiral for comparing data frames before and after derivations. It could print a few key pieces of information:

• Rows that match/don't match an optional ID
• Number of new/deleted row
• New/deleted columns
• Number of rows with NA for new variables

These are just a few thoughts and something we could do to modernize a log and functionalize it.

[bms63]

These might be kind of handy in the logrx package https://pharmaverse.github.io/logrx/ and printing messages to the log

[rossfarrugia]

Could use {diffdf} which already covers most of this.

Reading all the discussion here I'm tending towards Daniel's suggestion and not updating admiral here given the sizeable effort, and just providing users tips of how to use other packages/functions to be able to monitor differences between dataframes whilst you're developing your ADaM.

[rossfarrugia]

@manciniedoardo the above could even be a useful topic and experience share for a future admiral community meeting (depending on outcome of this discussion)

[bms63]

There is a admiral::adsl, logrx and diffdf example here - https://pharmaverse.github.io/logrx/articles/adsl_demo.html. It is very basic, but I use this wombo combo all the time at GSK.

[bundfussr]

What should be the purpose of the log?
Should it support debugging if something went wrong?
Should it document that everything went well?

[manciniedoardo]

ideally, all of the above!

[bundfussr]

I think for debugging running the code interactively step by step and examining the intermediate results is the easiest approach. (Whatever we write into the log in many cases it will not provide sufficient information to identify the issue.) The only case where this doesn't work is if an issue occurs in batch runs but not in interactive runs. For this case we could provide an option for batch runs which dumps the global environment into a file. Then the intermediate results from the batch run could be reviewed or compared with those from the interactive run.

For documenting that everything went well I wonder if we need more than the messages printed to the console (usually the functions called). Shouldn't no errors and warnings be sufficient to show that everything went well?

[manciniedoardo]

Thanks everyone for your inputs! In summary, it looks like the answer to the discussion question "Could admiral have a "verbose" mode where useful info is printed to the log after each function run?" seems to be "Maybe, but it would be very hard to implement and upkeep and it is unclear if the added benefit is worth it" 😆 .

Maybe for the next release of the package we can try brainstorm key thoughts from this discussion to identify where to provide targeted solutions, rather than go for a full SAS-style log approach to answer any questions. Ie, maybe it would be best to provide tips for users for how to check the status of datasets, as @rossfarrugia suggests, and/or explore how to integrate tools like diffdf in the workflow, similar to @ddsjoberg 's points.

[StefanThoma]

I was thinking of an additional function, a wrapper around diffdf and logrx or something, that you could just pipe into and then compares the original dataset to the output dataset. Would something like this work?

…

On Tue, May 7, 2024 at 2:28 PM Edoardo Mancini ***@***.***> wrote: Thanks everyone for your inputs! In summary, it looks like the answer to the discussion question *"Could admiral have a "verbose" mode where useful info is printed to the log after each function run?"* seems to be *"Maybe, but it would be very hard to implement and upkeep and it is unclear if the added benefit is worth it"* 😆 . Maybe for the next release of the package we can try brainstorm key thoughts from this discussion to identify where to provide targeted solutions, rather than go for a full SAS-style log approach to answer any questions. Ie, maybe it would be best to provide tips for users for how to check the status of datasets, as @rossfarrugia <https://github.com/rossfarrugia> suggests, and/or explore how to integrate tools like diffdf in the workflow, similar to @ddsjoberg <https://github.com/ddsjoberg> 's points. — Reply to this email directly, view it on GitHub <#2418 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJUWWEUFJ3OW2RAY3KW2SFTZBDCFBAVCNFSM6AAAAABGWVFIZCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNBRGE2DA> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.*** com>

[manciniedoardo]

Update: at today's {admiral} meeting @aefranklin from {roak} gave a super interesting demo showcasing how {roak} tackles messaging. Some notes from the demo and ensuing discussion (anyone feel free to add more):

• {roak} and {admiral} have different paradigms: while {roak} is a black-box automation solution, {admiral} is modular. With {roak}, users generate one whole SDTM dataset at a time and so need/benefit from messaging informing them about which variables are being derived, what order they are being derived in, how many rows the ensuing dataset has, etc. This helps debugging. This is less relevant for {admiral} because we have more open code and can run different blocks of code at different times or separately. Consequently messaging strategies may well differ across the two packages.
• Amy showcased some data checks that {roak} executes, e.g. comparing death date across to CRF pages. Agreement in the call was tat this is not something we want to do in general in {admiral}
• General consensus in the call was that we do not wish to implement enhanced logging or messaging capabilities in {admiral}. One proposal was to implement something outside of {admiral} that can be used to give info after a pipe chain is executed, similar to @StefanThoma 's suggestion above. @ddsjoberg noted this could then work with {dplyr} functions too. @sadchla-codes suggested integrating this into pharmaverse4devs.
• useful summary of changes would be: row count, variable count, variable values (e.g. all missing?)

[bms63]

I don't think logging should be apart of pharmaverse4devs as it is an addin/utility developer focused package that will never to go CRAN.

FYI You can use logrx or logr to do logging - https://pharmaverse.github.io/examples/logging/logging.html. I made an example that uses admiral on logrx site - https://pharmaverse.github.io/logrx/articles/adsl_demo.html