@ayferkrd it seems to be fine with me if I use below config. Do you see any difference?

resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit_percentage" {
    parent = "projects/myproject"
    description = "Description"
    display_name = "Displayname"

    triggers {
        schedule {
            recurrence_period_duration = "86400s"
        }
    }

    inspect_job {
        inspect_template_name = "fake"
        actions {
            save_findings {
                output_config {
                    table {
                        project_id = "project"
                        dataset_id = "dataset"
                    }
                }
            }
        }
        storage_config {
            big_query_options {
                table_reference {
                    project_id = "project"
                    dataset_id = "dataset"
                    table_id = "table_to_scan"
                }

                rows_limit_percent = 50
                sample_method = "RANDOM_START"
            }
            
            timespan_config {
                start_time = "2023-01-01T00:00:23Z"
                timestamp_field {
                    name = "timestamp"
                }
                enable_auto_population_of_timespan_config = true
            }
        }
    }

}

rows_limit_percent or rows_limit cannot be used with timespan_config:
Error: Error creating JobTrigger: googleapi: Error 400: Both TimespanConfig and RowsLimit may not be set at the same time.
│
│ with module.bq_dlp_trigger.google_data_loss_prevention_job_trigger.bigquery_row_limit_timestamp["test_table"],
│ on modules/bq_dlp_trigger/dlp_job_trigger.tf line 1, in resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit_timestamp":
│ 1: resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit_timestamp" {
│

@ayferkrd help me understand what the issue is. Below error comes from the api. Do you think this is wrong? The config works fine with me.

Error 400: Both TimespanConfig and RowsLimit may not be set at the same time

@edwardmedia above config has sample_method with rows_limit_percent with timespan_config which is not possible

@ayferkrd Interesting. I see what you mean. I am not able to hit the error with my config, which contradicts below api doc.
Wonder under what condition you can hit the error. Could you share your debug log?

b/275540727

https://cloud.google.com/dlp/docs/reference/rest/v2/InspectJobConfig#BigQueryOptions

@edwardmedia I think even it doesn't hit the error, I believe the behaviour is not correct with the above configuration because timestamp configuration cannot work with sampling, it should scan the whole table.

Create inspect job with both rows_limit and timespan_config:

• expected: fails
• actual: fails

Create inspect job with both rows_limit_percent and timespan_config:

• expected: fails
• actual: succeeds

Looks like this is an upstream API issue. We don't control this in the Terraform provider, and it seems like the documentation is wrong.

According to the docs, the API should be rejecting the request when both of timespan_config & rows_limit_percent are set, but they are incorrectly accepting it. We may be able to add ConflictsWith to block this on the provider side, but we probably don't want to make that decision for the API.

I'm going to mark this as upstream because the core of the issue is an API error

@edwardmedia, the real problem here is there is a default value set for sample_method, which blocks me from using timespan_config. Other than that, yes, both terraform and API has some problems independent of each other related to job trigger implementation.

b/280337903

I'm going to mark this as upstream because the core of the issue is an API error

Not entirely correct. As pointed out in #14096 (comment), this is an issue caused by the default value for sample_method which conflicts with Timespan filtering. Either setting timespan filtering should clear the sampling value, or the default for sample method should be unset.

I'm going to mark this as upstream because the core of the issue is an API error

Not entirely correct. As pointed out in #14096 (comment), this is an issue caused by the default value for sample_method which conflicts with Timespan filtering. Either setting timespan filtering should clear the sampling value, or the default for sample method should be unset.

I've come across this same issue where the provider is defaultiing sample_method to top within bigquery_options. In the meantime I'm just setting the default value for "rows_limit_percent" to 99 if rows_limit isn't defined as a sort of workaround for the issue of the provider defaulting to sample_method = "top" if we don't pass a value.

Quick question though, is it the case where we should open a new Issue for this, where it provides all of the details specific to the default value for sample_method? Or should we stick with this Issue for now?

I attempted to use a custom encoder to remove sample_method if timespan_config is set, which would be a workaround for this case, but my update tests won't pass as the test plan is not empty. I'm not sure when the default_value: :TOP gets applied, but ImportStateVerifyIgnore doesn't fix my test either — the plan still expects TOP.

I've done some additional testing though, and it looks like setting sample_method = "" will pass, without the drawback of the 99% row limit percent workaround. I'll update the documentation with an example using timespan_config with an empty string for sample_method and consider this fixed.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.