Large, below-quota deployments (<80 functions) failing with message "Build failed with status: EXPIRED..." #5967

trevor-rex · 2023-06-12T19:12:03Z

[REQUIRED] Environment info

firebase-tools: 12.3.1

Platform: macOS Monterey

[REQUIRED] Test case

Run firebase deploy --only functions with a codebase containing 61 2nd Cloud Functions and 1 1st gen Cloud Function.

[REQUIRED] Steps to reproduce

Same as test case

[REQUIRED] Expected behavior

I would expect this deployment to go through since it is below the deployment quota of 80 functions per minute.

[REQUIRED] Actual behavior

I saw that firebase-tools had a recent update to improve retry behavior for large deployments so I thought I would try it. I am still running into issues deploying under 80 (quota) functions, however. It seems like the deployment times out before all the function deployments go through.

Some of the functions fail to deploy with build with the following error:

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].

The build logs are empty and don't contain any useful information other than the build expiring.

This happens for about 20 of the 62 functions. Here is a truncated output of the full deploy:

=== Deploying to 'project'...

i  deploying functions
Running command: 
✔ Preparing the list of your Firebase projects

> lint
> eslint .


✔  functions: Finished running predeploy script.
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: Loaded environment variables from .env, .env.staging.
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (hidden MB) for uploading
i  functions: packaged functions (hidden MB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (1st Gen) function dailyBackup(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.

Functions deploy had errors with the following functions:
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
Function URL (functionName(us-central1)): redacted
Function URL (functionName(us-central1)): redacted
Function URL (functionName(us-central1)): redacted
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1

This is a small portion of the --debug log that printed for each failed function, but most of it did not seem useful:

  "original": {
    "name": "FirebaseError",
    "children": [],
    "exit": 1,
    "message": "Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted],
    "status": 3,
    "code": 3
  }

The text was updated successfully, but these errors were encountered:

trevor-rex · 2023-06-27T16:10:43Z

Hello, I just wanted to check in on this and see if any devs have any idea what may be causing it? It is currently blocking us from having a consistent CI/CD pipeline, since even manually breaking up and batching the deployments isn't a valid workaround. Thank you!

hunterua · 2023-06-30T14:03:51Z

Same issue there.... have 20-30 functions and always have deployment error on random functions.... same error
Build failed with status: EXPIRED and message: An unexpected error occurred.

Nushio · 2023-07-07T05:10:40Z

Saw #5969 and thought it might be related to this,

Line 368 of extensionsApi.ts states masterTimeout: 600000 (and several other operations do, too).

I tried changing this locally for my firebase-tools, but i'm still getting the same error, Expired..., when trying to deploy ~40 gen2 functions.

SikandAlex · 2023-07-07T22:41:57Z

Same issue. Limited (< 10) number of Python 2nd Gen Firebase functions. Very frustrating and limiting the ability to productionize.

I think it happened when I added a 5th or 6th function. My functions have a large number of dependencies so not sure if that affects the issue. I haven't investigated the size in detail but am actively working to minimize the size.

Additionally, I am using both JS and Python functions with the codebase configuration set to different values. I am in the process of migrating the only JS function over to Python. Despite my best efforts, I couldn't get CORS to work on the 2nd gen Node.js functions so I couldn't get my local front-end to connect to my deployed backend (non-emulated) so moving to Python which I have had success with (but this is an issue for a different post).

I can also confirm that all of the functions are updated on each deployment instead of only those that have changes. This seems to be the current behavior of using the codebase configuration as mentioned in a linked PR / issue. Once I remove the need for codebase I will update this post.

Edit: No change after migration to using no codebase parameter and only Python functions. Considering moving to Terraform based deployments.

hendrickson-tyler · 2023-08-09T18:59:49Z

Also been dealing with this issue. Ever since I updated my 30 or so cloud functions to v2, I've been noticing errors when deploying some functions (the problematic ones change every time):

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].
✔  functions[payment-deleteOldPayments(us-central1)] Successful update operation.
✔  functions[customer-deleteJobsOnCustomerDelete(us-central1)] Successful update operation.
✔  functions[hourEntry-sendHourEntryReminders(us-central1)] Successful update operation.
✔  functions[calendar-addEvent(us-central1)] Successful update operation.
Functions deploy had errors with the following functions:
        calendar-getEvents(us-central1)
        job-deletePaymentsOnJobDelete(us-central1)
        payment-deletePaymentCheckPhoto(us-central1)
        subcontractor-onSubcontractorDelete(us-central1)
        user-deleteTasksOnUserDelete(us-central1)
Function URL (user-registerDeviceToken(us-central1)): [redacted]
Function URL (user-createNewUser(us-central1)): [redacted]
Function URL (calendar-getEventsToday(us-central1)): [redacted]
Function URL (calendar-addEvent(us-central1)): [redacted]
Function URL (calendar-updateEvent(us-central1)): [redacted]
Function URL (calendar-deleteEvent(us-central1)): [redacted]
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function calendar-getEvents in region us-central1
- Error Failed to update function payment-deletePaymentCheckPhoto in region us-central1
- Error Failed to update function user-deleteTasksOnUserDelete in region us-central1
- Error Failed to update function subcontractor-onSubcontractorDelete in region us-central1
- Error Failed to update function job-deletePaymentsOnJobDelete in region us-central1
Warning: run-commands command "firebase deploy --config firebase.json" exited with non-zero status code

Even worse, when these functions fail to deploy, subsequent deploys tell me that there are "no changes detected". There definitely are since those changes never made it to my function to begin with:

=== Deploying to 'crown-finish-development'...
i  deploying storage, firestore, functions, hosting
i  firebase.storage: checking apps/back-end/storage.rules for compilation errors...
✔  firebase.storage: rules file apps/back-end/storage.rules compiled successfully
i  firestore: reading indexes from apps/back-end/firestore.indexes.json...
i  cloud.firestore: checking apps/back-end/firestore.rules for compilation errors...
✔  cloud.firestore: rules file apps/back-end/firestore.rules compiled successfully
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
i  functions: Loading and analyzing source code for codebase default to determine what to deploy
Serving at port 8191
shutdown requested via /__/quitquitquit
i  functions: preparing dist/apps/back-end directory for uploading...
i  functions: packaged /Users/tylerhendrickson/Developer/soluna-studios/crown-finish/dist/apps/back-end (91.11 KB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
i  storage: latest version of apps/back-end/storage.rules already up to date, skipping upload...
i  firestore: deploying indexes...
✔  firestore: deployed indexes in apps/back-end/firestore.indexes.json successfully for (default) database
i  firestore: latest version of apps/back-end/firestore.rules already up to date, skipping upload...
i  hosting[crown-finish-development]: beginning deploy...
i  hosting[crown-finish-development]: found 50 files in dist/apps/web
i  hosting: upload complete
✔  hosting[crown-finish-development]: file upload complete
✔  storage: released rules apps/back-end/storage.rules to firebase.storage
✔  firestore: released rules apps/back-end/firestore.rules to cloud.firestore
i  functions: Skipping the deploy of unchanged functions with experimental support for skipdeployingnoopfunctions
✔  functions[user-deleteTasksOnUserDelete(us-central1)] Skipped (No changes detected)
✔  functions[user-deleteHourEntriesOnUserDelete(us-central1)] Skipped (No changes detected)
✔  functions[user-registerDeviceToken(us-central1)] Skipped (No changes detected)
✔  functions[user-createNewUser(us-central1)] Skipped (No changes detected)
✔  functions[job-deletePaymentsOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteReceiptsOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteJobPhotosOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteHourEntriesOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteTasksOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[hourEntry-sendHourEntryReminders(us-central1)] Skipped (No changes detected)
✔  functions[task-sendNotificationOnAssignment(us-central1)] Skipped (No changes detected)
✔  functions[task-sendTaskReminders(us-central1)] Skipped (No changes detected)
✔  functions[payment-deletePaymentCheckPhoto(us-central1)] Skipped (No changes detected)
✔  functions[payment-deleteOldPayments(us-central1)] Skipped (No changes detected)
✔  functions[receipt-deleteReceipt(us-central1)] Skipped (No changes detected)
✔  functions[receipt-deleteOldReceipts(us-central1)] Skipped (No changes detected)
✔  functions[estimate-deleteJobPhotosOnEstimateDelete(us-central1)] Skipped (No changes detected)
✔  functions[estimate-sendEstimateReminders(us-central1)] Skipped (No changes detected)
✔  functions[jobPhoto-deleteJobPhoto(us-central1)] Skipped (No changes detected)
✔  functions[jobPhoto-deleteOldJobPhotos(us-central1)] Skipped (No changes detected)
✔  functions[customer-updateJobsOnCustomerRename(us-central1)] Skipped (No changes detected)
✔  functions[customer-updateEstimatesOnCustomerRename(us-central1)] Skipped (No changes detected)
✔  functions[customer-deleteJobsOnCustomerDelete(us-central1)] Skipped (No changes detected)
✔  functions[customer-deleteEstimatesOnCustomerDelete(us-central1)] Skipped (No changes detected)
✔  functions[subcontractor-onSubcontractorDelete(us-central1)] Skipped (No changes detected)
✔  functions[calendar-getEvents(us-central1)] Skipped (No changes detected)
✔  functions[calendar-getEventsToday(us-central1)] Skipped (No changes detected)
✔  functions[calendar-addEvent(us-central1)] Skipped (No changes detected)
✔  functions[calendar-updateEvent(us-central1)] Skipped (No changes detected)
✔  functions[calendar-deleteEvent(us-central1)] Skipped (No changes detected)
✔  functions[expiration-recordCreationTime(us-central1)] Skipped (No changes detected)
i  functions: cleaning up build files...
i  hosting[crown-finish-development]: finalizing version...
✔  hosting[crown-finish-development]: version finalized
i  hosting[crown-finish-development]: releasing new version...
✔  hosting[crown-finish-development]: release complete
✔  Deploy complete!

Now I can't trust my CI/CD to deploy reliably. Some changes make it while others don't until I deploy next time with new changes. This is a huge problem. Can we please have some movement on this?

hunterua · 2023-08-09T23:51:30Z

Any updates on this? Any estimations? 😳

Zheitk · 2023-08-13T21:18:08Z

I just tested with firebase tools 12.4.7 (I'm learning and already read all the documentation)
After reading some issues and pull request here, I was under the impression that the "skip no-op deploys" funcionality would skip the deploy of individual functions, but it seems to be based on whole codebases.

I have a project with a typescript "default" codebase, and a javascript "js" codebase, each one with 3 functions.
Changing even a character on any file of a codebase will cause firebase deploy --only functions to deploy ALL of the functions on said codebase, and skip only ALL the functions on the codebase that was unchanged.

Using only 1 codebase will thus trigger an update of all functions on the project after any change.

If the change is on 1 of 3 functions, and deployed with firebase deploy --only functions:function (or functions:codebase:function), firebase tools will deploy only said function, and then firebase deploy --only functions will skip that function (and the other codebase), but update the other 2 functions that were unchanged.

I read that firebase uploads and zips all of the codebase code for every function, so it seems that it compares the new zipped code with the uploaded code.

lox · 2023-08-15T06:18:23Z

We see this a lot too. It's very frustrating.

gillycheesesteak · 2023-08-24T18:44:22Z

Another bump for this, I hadn't realized that this was a problem until @hendrickson-tyler mentioned it

Even worse, when these functions fail to deploy, subsequent deploys tell me that there are "no changes detected". There definitely are since those changes never made it to my function to begin with

andrewkrippner · 2023-09-06T18:46:46Z

Huge problem for me as well!

gillycheesesteak · 2023-09-15T00:51:38Z

This seems to be happening less frequently for me the last couple of days, is anyone else seeing this? Did something change?

andrewkrippner · 2023-09-18T15:57:39Z

This seems to be happening less frequently for me the last couple of days, is anyone else seeing this? Did something change?

The frequency is dropping for me as well.

trevor-rex · 2023-09-18T16:29:17Z

This seems to be happening less frequently for me the last couple of days, is anyone else seeing this? Did something change?

The frequency is dropping for me as well.

I will say with the newer releases (in the past ~3 months) have seemed much smoother. Our CircleCI deployment success rate for ~60 functions is now 93%, and at the time of posting this issue it was below 50%.

gillycheesesteak · 2023-10-03T00:49:58Z

Any news on this? @Berlioz @colerogers sorry to tag you 🥺

joehan · 2023-10-03T17:41:19Z

@gillycheesesteak We're still working on ironing out some minor bugs with it, but #6376 should lead to some major improvements to this experience!

blidd-google · 2023-10-11T15:54:57Z

firebase-tools >=v12.6.1 includes the enabling of single builds for v2 functions, which should drastically improve deployment speed / reliability on large deployments. The feature should ameliorate the build expiring problem mentioned in this issue by reusing builds from the same deploy group. Will go ahead and close this thread for now, but please open a new issue if you encounter any other errors while deploying functions using the latest versions of the CLI tool.

cketcham · 2024-01-23T17:16:54Z

I'm experiencing this issue whenever all my functions have changed, but redeploying once seems to fix the issue since it only updates the ones that failed.

lordprana · 2024-02-23T01:46:37Z

I am still consistently experiencing this issue. With almost every deploy about 3-6 of my functions randomly fail to deploy with status "EXPIRED." Really need this to be fixed to be able to have a good time using Firebase Functions. This started for me in the past couple of weeks. Before that I was not having any problems.

lordprana · 2024-02-27T18:45:57Z

@blidd-google @joehan I am still consistently experiencing this issue deploying Cloud Functions V2 via Firebase. I'm using version 13.3.0 of the firebase-tools library, 4.4.1 of the firebase-functions library, and 11.11.1 of the firebase-admin library.

charifield · 2024-05-10T14:54:52Z

Any news on this? Still an issue for me, I have 120 functions, and 50% failure rate, every single deployment. PLEASE ASSIST @Berlioz @colerogers

SherpaMiguel · 2024-05-11T14:46:44Z

This issue is still a nightmare when deploying on production.

Mashoodx · 2024-05-13T12:04:44Z

still happening guys

guillermodlpa · 2024-05-13T13:09:57Z

I'm also experiencing this error frequently in the last few days

kostiantyn-pravdiuk · 2024-05-13T13:11:34Z

I have the same problem

SherpaMiguel · 2024-05-13T13:11:41Z

I think this issue was reintroduced in 13.8.1. After downgrading to 13.8.0, errors were gone.

hendrickson-tyler · 2024-05-14T03:23:54Z

Yep, can confirm. I'm seeing this issue again on firebase-tools v13.8.3. Absolutely maddening.

Downgrading to v13.8.0 as @SherpaMiguel suggested did seem to work for me. Might want to pin to that version until this issue is fixed again...

arnoutvandervorst · 2024-05-14T09:53:01Z

Since v13.8.1, the optimizations have been disabled and made available as experiment:

firebase experiments:enable functionsv2deployoptimizations

kostiantyn-pravdiuk · 2024-05-15T12:51:42Z

@arnoutvandervorst thanks it works

joehan · 2024-05-15T15:31:10Z

Sorry for the inconvience everyone. In 13.8.1, we disabled the feature that fixed this issue to mitigate a backend bug that affected some regions. We're planning on reenabling it soon once we have confirmation that the backend bug is fixed.

In the interim, if you're having this issue and needed to downgrade, instead try enabling the experiment they are behind:

firebase experiments:enable functionsv2deployoptimizations

rosshabe · 2024-05-16T13:34:25Z

Worked for me thanks

tbusot · 2024-05-22T14:04:46Z

Fix worked for us as well.

@joehan How will we (users) be alerted about a permanent fix? I'm injecting the experiments:enable tag into my github actions and will want to know when to remove it.
Is it just a matter of watching the change logs? This ticket is already closed, so not sure that re-closing it will give us the clue. Thanks!

toddmotto · 2024-05-22T21:09:10Z

Seeing this all day today, have just ran the suggested fix for firebase experiments:enable functionsv2deployoptimizations and it did work.

While I'm here, I'd also like to say it's a nightmare using the new V2 functions sometimes they just seem to 'corrupt' and I have to manually delete all the functions via the CLI, then likely have to reinstate the IAM permissions in the cloud console for every single function, then sometimes the V2 functions don't deploy properly and tell me I'm changing to the new format upon first deploys, likely due to me mixing V1 and V2 functions (because I have to for V1 functions.auth.user().onCreate hooks etc.

charifield · 2024-06-01T06:09:37Z

Man, this is still happening to me. Tried the firebase experiments:enable functionsv2deployoptimizations and still happens. Any other fixes in place for this?

SherpaMiguel · 2024-06-01T15:17:46Z

It started happening again in 13.10.2. Now even downgrading to 13.8.0 fails. Looks like functionsv2deployoptimizations is disabled behind the scenes.

chriswaterbury · 2024-06-02T18:00:24Z

I'm also now seeing this even after running firebase experiments:enable functionsv2deployoptimizations.

ceifa · 2024-06-03T14:32:28Z

@trevor-rex @blidd-google Can we reopen this issue as it is still a problem?

charifield · 2024-06-03T14:35:37Z

@Berlioz @colerogers Definitely an issue. Has gotten worse, deployments are consistently failing. Please assist in reopening issue

trevor-rex · 2024-06-03T16:11:01Z

@trevor-rex @blidd-google Can we reopen this issue as it is still a problem?

I'm unable to re-open the issue but hopefully someone from Google does. I have never upgraded from 13.8.0 as I saw that the deployment issue was reintroduced in 13.8.1, and I am not having issues with deployments. I am not sure if never having upgraded or not enabling the experiment was the reason for this. I hope that someone from Google can re-enable single builds for v2 functions as soon as possible, as my team will be freezing our firebase-tools version at 13.8.0 until this is resolved.

wojtek-iwarranty · 2024-06-04T05:30:35Z

Hi, before weekend it happened just from time to time. Now it happens all the time for all our projects. No solution described here works for us. We are blocked...

gersonlimadev · 2024-06-04T12:40:57Z

It worked normally last week but now the functions are failing every deployment. I've tried rollback to the previous version but it still fails.

trevor-rex · 2024-06-04T15:52:08Z

It may be resolved the quickest if someone opens a new Github issue, as re-opening this ticket doesn't seem to be urgent for the repo owners. I am unable to do this myself as I don't have any error logs to provide to the Google team since I'm not encountering the issue.

wojtek-iwarranty · 2024-06-04T19:11:59Z

New issue opened #7268

yeonjoonkim · 2024-06-17T10:14:43Z

I am getting the same error but I created a script to deploy individual functions one by one but there is no efficiency...
I am wondering has Anyone solved this issue?

Deployed 27 functions successfully
Failed to deploy 0 functions
Total time taken: 55 min 58.79 sec
Deployed 27 functions successfully (100.00%)
Failed to deploy 0 functions (0.00%)

`import { TriggerList } from './function-name/trigger-name.mjs';
import { SchedulerList } from './function-name/scheduler-name.mjs';
import { APIList } from './function-name/api-name.mjs';
import { execSync } from 'child_process';
import readlineSync from 'readline-sync';

const candidates = [...TriggerList, ...SchedulerList, ...APIList];

const debugOption = readlineSync.question(Do you want to deploy in debug mode? [Y/N] (Default: N)\n);
let isDebugMode = false;

if (debugOption === 'Y' || debugOption === 'y') {
isDebugMode = true;
}

const start = new Date();

deployCmds.forEach((cmd, index) => {
try {
const startDateTime = new Date();
console.log(Deploying ${index + 1}/${length} => ${cmd});
execSync(cmd, { stdio: 'inherit' });
const endDateTime = new Date();
const duration = endDateTime - startDateTime;
const minutes = Math.floor(duration / 60000);
const seconds = ((duration % 60000) / 1000).toFixed(2);
console.log(Total time taken: ${minutes} min ${seconds} sec);
console.log(---------------------);
success++;
} catch (error) {
console.error(Error executing command: ${cmd}, error);
console.log(---------------------);
failed++;
}
});

const end = new Date();
const totalDuration = end - start;
const totalMinutes = Math.floor(totalDuration / 60000);
const totalSeconds = ((totalDuration % 60000) / 1000).toFixed(2);
const successPercentage = ((success / length) * 100).toFixed(2);
const failurePercentage = ((failed / length) * 100).toFixed(2);

console.log(Deployed ${success} functions successfully);
console.log(Failed to deploy ${failed} functions);
console.log(Total time taken: ${totalMinutes} min ${totalSeconds} sec);
console.log(Deployed ${success} functions successfully (${successPercentage}%));
console.log(Failed to deploy ${failed} functions (${failurePercentage}%));
console.log(---------------------);
`

trevor-rex added the type: bug label Jun 12, 2023

trevor-rex changed the title ~~Large deployments (<80) failing with message "Build failed with status: EXPIRED..."~~ Large, below-quota deployments (<80 functions) failing with message "Build failed with status: EXPIRED..." Jun 12, 2023

aalej added the api: functions label Jun 13, 2023

joehan assigned colerogers Jun 13, 2023

gillycheesesteak mentioned this issue Jun 13, 2023

Do not re-deploy unchanged functions when using --only functions:codebase #5959

Open

joehan assigned Berlioz Jun 27, 2023

joehan assigned blidd-google and unassigned Berlioz and colerogers Jul 11, 2023

joehan assigned colerogers Aug 15, 2023

aalej assigned Berlioz Aug 29, 2023

blidd-google closed this as completed Oct 11, 2023

wojtek-iwarranty mentioned this issue Jun 4, 2024

deployments (55 functions) failing with message "Build failed with status: EXPIRED..." #7268

Closed

berengamble mentioned this issue Jul 10, 2024

4 out of 13 builds fail with: Build failed with status: EXPIRED and message: An unexpected error occurred #7430

Open

Large, below-quota deployments (<80 functions) failing with message "Build failed with status: EXPIRED..." #5967

Large, below-quota deployments (<80 functions) failing with message "Build failed with status: EXPIRED..." #5967

Comments

trevor-rex commented Jun 12, 2023

[REQUIRED] Environment info

[REQUIRED] Test case

[REQUIRED] Steps to reproduce

[REQUIRED] Expected behavior

[REQUIRED] Actual behavior

trevor-rex commented Jun 27, 2023

hunterua commented Jun 30, 2023

Nushio commented Jul 7, 2023

SikandAlex commented Jul 7, 2023 • edited Loading

hendrickson-tyler commented Aug 9, 2023

hunterua commented Aug 9, 2023

Zheitk commented Aug 13, 2023

lox commented Aug 15, 2023

gillycheesesteak commented Aug 24, 2023

andrewkrippner commented Sep 6, 2023

gillycheesesteak commented Sep 15, 2023

andrewkrippner commented Sep 18, 2023

trevor-rex commented Sep 18, 2023

gillycheesesteak commented Oct 3, 2023

joehan commented Oct 3, 2023

blidd-google commented Oct 11, 2023

cketcham commented Jan 23, 2024

lordprana commented Feb 23, 2024

lordprana commented Feb 27, 2024

charifield commented May 10, 2024 • edited Loading

SherpaMiguel commented May 11, 2024

Mashoodx commented May 13, 2024

guillermodlpa commented May 13, 2024

kostiantyn-pravdiuk commented May 13, 2024

SherpaMiguel commented May 13, 2024 • edited Loading

hendrickson-tyler commented May 14, 2024

arnoutvandervorst commented May 14, 2024

kostiantyn-pravdiuk commented May 15, 2024

joehan commented May 15, 2024

rosshabe commented May 16, 2024

tbusot commented May 22, 2024

toddmotto commented May 22, 2024

charifield commented Jun 1, 2024

SherpaMiguel commented Jun 1, 2024 • edited Loading

chriswaterbury commented Jun 2, 2024 • edited Loading

ceifa commented Jun 3, 2024

charifield commented Jun 3, 2024 • edited Loading

trevor-rex commented Jun 3, 2024

wojtek-iwarranty commented Jun 4, 2024

gersonlimadev commented Jun 4, 2024

trevor-rex commented Jun 4, 2024 • edited Loading

wojtek-iwarranty commented Jun 4, 2024 • edited Loading

yeonjoonkim commented Jun 17, 2024

Deployed 27 functions successfully Failed to deploy 0 functions Total time taken: 55 min 58.79 sec Deployed 27 functions successfully (100.00%) Failed to deploy 0 functions (0.00%)

SikandAlex commented Jul 7, 2023 •

edited

Loading

charifield commented May 10, 2024 •

edited

Loading

SherpaMiguel commented May 13, 2024 •

edited

Loading

SherpaMiguel commented Jun 1, 2024 •

edited

Loading

chriswaterbury commented Jun 2, 2024 •

edited

Loading

charifield commented Jun 3, 2024 •

edited

Loading

trevor-rex commented Jun 4, 2024 •

edited

Loading

wojtek-iwarranty commented Jun 4, 2024 •

edited

Loading

Deployed 27 functions successfully
Failed to deploy 0 functions
Total time taken: 55 min 58.79 sec
Deployed 27 functions successfully (100.00%)
Failed to deploy 0 functions (0.00%)