Merge remote-tracking branch 'origin/main'

.gitignore

@@ -3,7 +3,9 @@
 *.pt
 
 # Credential files
+evaluation_examples/settings/google/settings.json
 evaluation_examples/settings/googledrive/credentials.json
+evaluation_examples/settings/googledrive/client_secrets.json
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

ACCOUNT_GUIDELINE.md

@@ -0,0 +1,138 @@
+# Real Accounts
+
+For tasks including google or google drive, we need a real Google account as well as configured OAuth2.0 secrets. 
+
+> Attention: to prevent environment reset and result evaluation conflicts caused by multiple people using the same Google account simultaneously, we will not provide the public test accounts available. Please register a private Google account.
+
+## Table of Contents
+1. [Register A Blank Google Account](#register-a-blank-google-account)
+2. [Create A Google Cloud Project](#create-a-google-cloud-project)
+3. [Configure OAuth Consent Screen](#configure-oauth-consent-screen)
+4. [Create OAuth2.0 Credentials](#create-oauth20-credentials)
+5. [Potential Issues](#potential-issues)
+
+## Register A Blank Google Account
+
+1. Go to Google web site and register a blank new account
+  - In this testbed, you do not need to provide any recovery email or phone, since we only use it for testing cases
+  - Just **IGNORE** any security recommendations
+  - Shut **OFF** the [2-Step Verification](https://support.google.com/accounts/answer/1064203?hl=en&co=GENIE.Platform%3DDesktop#:~:text=Open%20your%20Google%20Account.,Select%20Turn%20off.) to avoid failure in environment setup (requesting phone verification code)
+
+<p align="center">
+  <img src="assets/googleshutoff.png" width="40%" alt="Shut Off 2-Step Verification">
+</p>
+
+> Attention: we strongly recommend that you register a new blank account instead of using an existing one, in order to avoid messing up your personal workspace.
+
+2. Next, copy and rename the template file `settings.json.template` into `settings.json` under folder `evaluation_examples/settings/google/`. Remember to replace the two fields `email` and `password`:
+  - these two fields are used to simulate real people login to Chrome browser during environment setup for relevant examples in the virtual machine
+
+```
+{
+    "email": "[email protected]",
+    "password": "your_google_account_password"
+}
+```
+
+## Create A Google Cloud Project
+
+1. Navigate to [Google Cloud Project Creation](https://console.cloud.google.com/projectcreate) page and create a new GCP (see [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project) for detailed steps). You can use any project name.
+
+2. Go to the [Google Drive API console](https://console.cloud.google.com/apis/library/drive.googleapis.com?) and enable the GoogleDrive API for the created project (see [Enable and disable APIs](https://support.google.com/googleapi/answer/6158841?hl=en) for detailed steps)
+
+<p align="center">
+  <img src="assets/creategcp.png" width="45%" style="margin-right: 5%;" alt="Create GCP">
+  <img src="assets/enableapi.png" width="45%" alt="Google Drive API">
+</p>
+
+## Configure OAuth Consent Screen
+To configure the OAuth2.0 screen for the created GCP. Go to page [OAuth consent screen](https://console.cloud.google.com/apis/credentials/consent?):
+
+1. For User Type, choose "External" and click "Create"
+
+<p align="center">
+  <img src="assets/usertype.png" width="50%" alt="User Type">
+</p>
+
+2. For App information, type in any App name you like (e.g., DataAccess), and choose the current Google gmail into field `User support email`.
+
+<p align="center">
+  <img src="assets/oauthapp.png" width="50%" alt="App Info">
+</p>
+
+3. For Developer information, also fill in the current gmail account. Leave other fields blank and click button "SAVE AND CONTINUE".
+
+<p align="center">
+  <img src="assets/developer.png" width="50%" alt="Developer information">
+</p>
+
+4. Leave fields blank for `Scopes` and continue to `Test Users`. Add the current gmail account via clicking button "+ ADD USERS".
+
+<p align="center">
+  <img src="assets/testusers.png" width="50%" alt="Test Users">
+</p>
+
+5. Finish all configuration and we will come to the configured OAuth consent screen. There is another thing, PUBLISH APP to extend the lifecycle of credentials. Otherwise, the refresh token is only valid in 7 days (refer to [google offical doc](https://developers.google.com/identity/protocols/oauth2#expiration) and [stackoverflow post](https://stackoverflow.com/questions/74659774/google-oauth-2-0-refresh-token-expiry-how-many-days) for details).
+
+<p align="center">
+  <img src="assets/publishapp.png" width="50%" alt="Publish APP">
+</p>
+
+
+## Create OAuth2.0 Credentials
+
+1. Goto the [credentials page](https://console.cloud.google.com/apis/credentials?), click "CREATE CREDENTIALS -> OAuth client ID"
+
+<p align="center">
+  <img src="assets/oauth2.0.png" width="50%" alt="Create OAuth client ID">
+</p>
+
+2. For Application type, please choose "Desktop app". You can use any Name. And click "CREATE".
+
+<p align="center">
+  <img src="assets/desktopapp.png" width="50%" alt="Desktop App">
+</p>
+
+3. Now, in the pop-up window, you can download the JSON file `client_secret_xxxxx.json`. Move and rename this .json file to file path `evaluation_examples/settings/googledrive/client_secrets.json` in the OSWorld project. The folder should look like:
+
+```
+- evaluation_examples/
+  - settings/
+    - google/
+      - settings.json
+      - settings.json.template
+    - googledrive/
+      - settings.yml
+      - client_secrets.json
+```
+
+4. Note that, when you first run a task including Google Drive, there will be a url requesting your permission. Open the link in unsafe mode using the gmail you filled in `evaluation_examples/settings/google/settings.json`, authorize and confirm your choice once for all. Eventually, you will see a prompt message "The authentication flow has completed." in a blank web page.
+
+<p align="center">
+  <img src="assets/unsafemode.png" width="45%" style="margin-right: 5%;" alt="Unsafe mode">
+  <img src="assets/authorization.png" width="45%" alt="Authorization">
+</p>
+
+## Potential Issues
+
+Due to strict check by Google safety teams, even if we shut down the 2-step verification, Google still detects potential risks of your account, especially __when you frequently change the login device__. You may encounter the following issues:
+
+### Phone Verification Code Required
+
+When the VM tries to log into the Google Drive page, Google requests you to provide a phone number and verfification number. This may occur when you change your IP or device for the first time.
+
+<p align="center">
+  <img src="assets/googlephonecode.png" width="50%" alt="Phone Verification Code Required">
+</p>
+
+To solve it, typing any phone number is adequate (since we shut off the 2-step verification and do not provide any recovery phone number). And fill in the received verification code. After that, hopefully, Google will remember this new login IP or device. Now, you can restart the task, and in this time, it should work.
+
+### Identity Verification
+
+<p align="center">
+  <img src="assets/googleidentity.png" width="50%" alt="Identity Verification">
+</p>
+
+In this case, Google does not give you the chance to use phone verification code. Since we do not provide any recovery email/phone and shut down the 2-step verification, we are unable to login from the new device. We hypothesize that this problem may occur when you frequently change the login IPs or devices, such that Google detects the unusual usages. The only solution is to reset the password from the device in which you register this Google account.
+
+> Sadly, we do not have a permanent solution. The only suggestion is not to frequently change your login IP or device. If you encounter any problem above, Google may urge you to change the password. Also remember to update the password in `evaluation_examples/settings/google/settings.json`.

README.md

@@ -140,6 +140,10 @@ Afterward, you can execute a command similar to the one in the previous section
 The username and password for the virtual machines are as follows:
 - **Ubuntu:** `user` / `password`
 
+### How to setup the account and credentials for Google and Google Drive?
+
+See [Account Guideline](ACCOUNT_GUIDELINE.md)
+
 ### How can I configure a proxy for the VM if I'm behind a GFW?
 
 See [Proxy Guideline](PROXY_GUIDELINE.md).

evaluation_examples/examples/thunderbird/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json

@@ -1,7 +1,7 @@
 {
   "id": "7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3",
   "snapshot": "thunderbird",
-  "instruction": "Help me access my outlook account with address \"[email protected]\" and password 'gTCI\";=@y7|QJ0nDa_kN3Sb&>' (without ')",
+  "instruction": "Help me access my outlook account with address \"[email protected]\" and password 'Wlv(z._6y|a,rrjfQuQhIi\$;' (without ')",
   "source": "https://www.wikihow.com/Access-Gmail-With-Mozilla-Thunderbird",
   "config": [
     {
@@ -86,12 +86,12 @@
           {
             "url": "imap:https://outlook.office365.com",
             "user": "[email protected]",
-            "password": "gTCI\";=@y7|QJ0nDa_kN3Sb&>"
+            "password": "Wlv(z._6y|a,rrjfQuQhIi\$;"
           },
           {
             "url": "smtp:https://smtp.office365.com",
             "user": "[email protected]",
-            "password": "gTCI\";=@y7|QJ0nDa_kN3Sb&>"
+            "password": "Wlv(z._6y|a,rrjfQuQhIi\$;"
           }
         ]
       }

evaluation_examples/settings/google/settings.json.template

@@ -0,0 +1,4 @@
+{
+    "email": "[email protected]",
+    "password": "your_google_account_password"
+}