-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DLP job trigger on bigquery cannot use timespan config #14096
DLP job trigger on bigquery cannot use timespan config #14096
Comments
@ayferkrd it seems to be fine with me if I use below config. Do you see any difference? resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit_percentage" {
parent = "projects/myproject"
description = "Description"
display_name = "Displayname"
triggers {
schedule {
recurrence_period_duration = "86400s"
}
}
inspect_job {
inspect_template_name = "fake"
actions {
save_findings {
output_config {
table {
project_id = "project"
dataset_id = "dataset"
}
}
}
}
storage_config {
big_query_options {
table_reference {
project_id = "project"
dataset_id = "dataset"
table_id = "table_to_scan"
}
rows_limit_percent = 50
sample_method = "RANDOM_START"
}
timespan_config {
start_time = "2023-01-01T00:00:23Z"
timestamp_field {
name = "timestamp"
}
enable_auto_population_of_timespan_config = true
}
}
}
} |
rows_limit_percent or rows_limit cannot be used with timespan_config: |
@edwardmedia above config has sample_method with rows_limit_percent with timespan_config which is not possible |
@ayferkrd Interesting. I see what you mean. I am not able to hit the error with my config, which contradicts below api doc. b/275540727 https://cloud.google.com/dlp/docs/reference/rest/v2/InspectJobConfig#BigQueryOptions |
@edwardmedia I think even it doesn't hit the error, I believe the behaviour is not correct with the above configuration because timestamp configuration cannot work with sampling, it should scan the whole table. |
Create inspect job with both rows_limit and timespan_config:
Create inspect job with both rows_limit_percent and timespan_config:
|
Looks like this is an upstream API issue. We don't control this in the Terraform provider, and it seems like the documentation is wrong. According to the docs, the API should be rejecting the request when both of timespan_config & rows_limit_percent are set, but they are incorrectly accepting it. We may be able to add ConflictsWith to block this on the provider side, but we probably don't want to make that decision for the API. I'm going to mark this as upstream because the core of the issue is an API error |
@edwardmedia, the real problem here is there is a default value set for sample_method, which blocks me from using timespan_config. Other than that, yes, both terraform and API has some problems independent of each other related to job trigger implementation. |
b/280337903 |
Not entirely correct. As pointed out in #14096 (comment), this is an issue caused by the default value for |
I've come across this same issue where the provider is defaultiing Quick question though, is it the case where we should open a new Issue for this, where it provides all of the details specific to the default value for |
I attempted to use a custom encoder to remove I've done some additional testing though, and it looks like setting |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Community Note
modular-magician
user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned tohashibot
, a community member has claimed the issue already.Terraform Version
Terraform v1.4.2
on darwin_amd64
Affected Resource(s)
Terraform Configuration Files
Debug Output
Error: Error creating JobTrigger: googleapi: Error 400:
rows_limit
orrows_limit_percent
must be greater than zero when sample method is specified.│
│ with module.bq_dlp_trigger.google_data_loss_prevention_job_trigger.bigquery_row_limit_timestamp,
│ on modules/bq_dlp_trigger/dlp_job_trigger.tf line 1, in resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit_timestamp":
│ 1: resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit_timestamp" {
│
Panic Output
Expected Behavior
Should create a DLP job trigger with timestamp configuration to run through the BigQuery.
Actual Behavior
Cannot create DLP job trigger
Steps to Reproduce
terraform apply
Important Factoids
rows_limit or rows_limit_percent cannot be used in conjunction with TimespanConfig.
References
[<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor documentation? For example:
--->](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/data_loss_prevention_job_trigger#attributes-reference)
The text was updated successfully, but these errors were encountered: