Run Flank in GitHub Action #2304

vorburger · 2023-10-26T08:09:51Z

I would like to explore if it's possible to run (trigger?) Flank (https://flank.github.io/flank/) in the GitHub Action (which now works fine otherwise),

i.e. ./gradlew :engine:runFlank etc.

as in https://github.com/google/android-fhir/blob/master/kokoro/gcp_ubuntu/kokoro_build.sh

If this works, then I'm not sure this project still needs the Google internal only Kokoro infrastructure? (Thus "fixing" e.g. #2300 or #2297 et al.)

This would, to me, lead to a simplification of this project's (to me, seemingly) "complex" CI infra.

The text was updated successfully, but these errors were encountered:

omarismail94 · 2023-10-26T17:38:45Z

runFlank runs the Firebase tests. We need a way to authenticate with Firebase if we run things in GitHub, and the only way I can think of is using a JSON key. Exporting a JSON key is not allowed and Im not sure how difficult it is to get an exception. Why this works in Kokoro is becasue it runs on GCP, which uses IAM roles to authenticate as opposed to JSON keys

vorburger · 2023-10-26T18:31:52Z

Exporting a JSON key is not allowed and Im not sure how difficult it is to get an exception.

I'll reach out privately because I would like to learn more about this...

vorburger · 2023-10-26T19:11:46Z

OK, I'm learning more about this, so there are 1-2 problems, here:

This projects' Firebase GCP project has Org restrictions which make it impossible to export Service Account credentials as a JSON file (which I would then securely store in a GitHub Secret). There is a process around it; I'll try to find out more about if this may be feasible.
Required Firebase Roles aren't really clear to me, yet.

vorburger · 2023-10-27T20:25:40Z

Note to future self: The following is just a Warning and it already always appears and is no problem:

$ ./gradlew :engine:runFlank

> Task :engine:execFlank
version: v21.11.0
revision: ...
session id: ...

Warning: Failed to make bucket for android-fhir-instrumeted-tests
Cause: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

This is NOT related to and not NOT the root cause of problems such as the following:

ftl.run.exception.PermissionDenied: Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://testing.googleapis.com/v1/testEnvironmentCatalog/ios?projectId=android-fhir-instrumeted-tests
{
  "code": 403,
  "errors": [
    {
      "domain": "global",
      "message": "Not authorized for project android-fhir-instrumeted-tests",
      "reason": "forbidden"
    }
  ],
  "message": "Not authorized for project android-fhir-instrumeted-tests",
  "status": "PERMISSION_DENIED"
}

vorburger · 2023-10-27T21:22:51Z

The 403 above is fixed by granting the "Editor" role to the SA, as per https://firebase.google.com/docs/test-lab/android/continuous. (Note to future self: It seemed like that it didn't seem to work when I added the "Editor" role to an existing SA, but worked if I created a new one to which I added it at creation time? I'm probably just very confused about the GCP Console; using gcloud CLI next time around may be clearer.)

With this I'm half way there, but still fails at the end:

$ ./gradlew :engine:runFlank

> Task :engine:execFlank
version: v21.11.0
revision: 1bf9b6a8ed1287141b66a75ec66b23813a02f2a8
session id: 9d601213-7b90-4cd5-a1f8-18168dcf3916

Warning: Failed to make bucket for android-fhir-instrumeted-tests
Cause: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
AndroidArgs
    gcloud:
      results-bucket: android-fhir-build-artifacts
      results-dir: engine-d42a277b-3f3b-4cd8-8ddc-98a5303f51ad
      record-video: true
      timeout: 45m
      async: false
      client-details:
      network-profile: null
      results-history-name: null
      # Android gcloud
      app: /home/vorburger/git/github.com/google/android-fhir/demo/build/outputs/apk/androidTest/debug/demo-debug-androidTest.apk
      test: /home/vorburger/git/github.com/google/android-fhir/engine/build/outputs/apk/androidTest/debug/engine-debug-androidTest.apk
      additional-apks:
      auto-google-login: false
      use-orchestrator: true
      directories-to-pull:
        - /sdcard/Download
      grant-permissions: all
      type: null
      other-files:
      scenario-numbers:
      scenario-labels:
      obb-files:
      obb-names:
      performance-metrics: true
      num-uniform-shards: null
      test-runner-class: null
      test-targets:
      robo-directives:
      robo-script: null
      device:
        - model: Nexus6P
          version: 24
          locale: en_US
          orientation: portrait
        - model: MediumPhone.arm
          version: 33
          locale: en_US
          orientation: portrait
      num-flaky-test-attempts: 1
      test-targets-for-shard:
      fail-fast: false
      parameterized-tests: default

    flank:
      max-test-shards: 10
      shard-time: -1
      num-test-runs: 1
      smart-flank-gcs-path: 
      smart-flank-disable-upload: false
      default-test-time: 120.0
      use-average-test-time-for-new-tests: false
      files-to-download:
      test-targets-always-run:
      disable-sharding: false
      project: android-fhir-instrumeted-tests
      local-result-dir: results
      full-junit-result: false
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
      run-timeout: -1
      legacy-junit-result: false
      ignore-failed-tests: false
      output-style: single
      disable-results-upload: false
      default-class-test-time: 240.0
      disable-usage-statistics: false
      output-report: none
      skip-config-validation: false
      custom-sharding-json: 

RunTests

 Smart Flank cache hit: 0% (0 / 20)
  Shard times: 240s, 240s, 240s, 240s, 240s, 240s, 240s, 240s, 240s, 360s

  Saved 1 shards to /home/vorburger/git/github.com/google/android-fhir/engine/build/fladle/results/engine-d42a277b-3f3b-4cd8-8ddc-98a5303f51ad/android_shards.json
  Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/android-fhir-build-artifacts/engine-d42a277b-3f3b-4cd8-8ddc-98a5303f51ad/...

Error on uploading android_shards.json
Cause: com.google.cloud.storage.StorageException: 403 Forbidden
POST https://storage.googleapis.com/upload/storage/v1/b/android-fhir-build-artifacts/o?projection=full&uploadType=multipart
{
  "error": {
    "code": 403,
    "message": "vorburger-test3@android-fhir-instrumeted-tests.iam.gserviceaccount.com does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).",
    "errors": [
      {
        "message": "vorburger-test3@android-fhir-instrumeted-tests.iam.gserviceaccount.com does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).",
        "domain": "global",
        "reason": "forbidden"
      }
    ]
  }
}


Total run duration: 0m  9s
        - Preparation: 0m 4s

> Task :engine:execFlank FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':engine:execFlank'.
> Process 'command '/usr/lib/jvm/java-17-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 11s
8 actionable tasks: 3 executed, 5 up-to-date

This is strange, because the Editor role should have the storage.objects.create permission?

vorburger · 2023-10-27T21:24:31Z

The (2nd) 403 above is because there are actually 2 separate GCP projects involved here:

android-fhir-instrumeted-tests and android-fhir-build (and the android-fhir-build-artifacts GCS bucket is in the latter)

I'll follow up internally at work to get access sorted out.

vorburger added help wanted Extra attention is needed type:testing Improvements on tests or testing infrastructure P2 Medium priority issue effort:medium Medium effort - 3 to 5 days type:build Issues related to code build labels Oct 26, 2023

vorburger self-assigned this Oct 26, 2023

vorburger mentioned this issue Oct 26, 2023

Kokoro Check build failures due to NodeJS or OpenJDK installation failure #2297

Open

vorburger added a commit that referenced this issue Oct 27, 2023

Add runFlank to GitHub Build Action (fixes #2304)

e7318aa

vorburger linked a pull request Oct 27, 2023 that will close this issue

Add runFlank to GitHub Build Action (fixes #2304) #2321

Draft

vorburger mentioned this issue Oct 27, 2023

Switch Firebase Test Lab Artifacts to another GCP GCS Bucket #2322

Closed

vorburger added a commit that referenced this issue Oct 27, 2023

Add runFlank to GitHub Build Action (fixes #2304)

c3a8e9c

This was referenced Oct 27, 2023

Factor out runFlank (re. #2321) #2323

Merged

Kokoro: Build and Device Tests Expected — Waiting for status to be reported #2300

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run Flank in GitHub Action #2304

Run Flank in GitHub Action #2304

vorburger commented Oct 26, 2023

omarismail94 commented Oct 26, 2023

vorburger commented Oct 26, 2023

vorburger commented Oct 26, 2023 •

edited

Loading

vorburger commented Oct 27, 2023

vorburger commented Oct 27, 2023

vorburger commented Oct 27, 2023 •

edited

Loading

Run Flank in GitHub Action #2304

Run Flank in GitHub Action #2304

Comments

vorburger commented Oct 26, 2023

omarismail94 commented Oct 26, 2023

vorburger commented Oct 26, 2023

vorburger commented Oct 26, 2023 • edited Loading

vorburger commented Oct 27, 2023

vorburger commented Oct 27, 2023

vorburger commented Oct 27, 2023 • edited Loading

vorburger commented Oct 26, 2023 •

edited

Loading

vorburger commented Oct 27, 2023 •

edited

Loading