Large, below-quota deployments (<80 functions) failing with message "Build failed with status: EXPIRED..."

[trevor-rex]

[REQUIRED] Environment info

firebase-tools: 12.3.1

Platform: macOS Monterey

[REQUIRED] Test case

Run firebase deploy --only functions with a codebase containing 61 2nd Cloud Functions and 1 1st gen Cloud Function.

[REQUIRED] Steps to reproduce

Same as test case

[REQUIRED] Expected behavior

I would expect this deployment to go through since it is below the deployment quota of 80 functions per minute.

[REQUIRED] Actual behavior

I saw that firebase-tools had a recent update to improve retry behavior for large deployments so I thought I would try it. I am still running into issues deploying under 80 (quota) functions, however. It seems like the deployment times out before all the function deployments go through.

Some of the functions fail to deploy with build with the following error:

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].

The build logs are empty and don't contain any useful information other than the build expiring.

This happens for about 20 of the 62 functions. Here is a truncated output of the full deploy:

=== Deploying to 'project'...

i  deploying functions
Running command: 
✔ Preparing the list of your Firebase projects

> lint
> eslint .


✔  functions: Finished running predeploy script.
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: Loaded environment variables from .env, .env.staging.
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (hidden MB) for uploading
i  functions: packaged functions (hidden MB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (1st Gen) function dailyBackup(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.

Functions deploy had errors with the following functions:
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
Function URL (functionName(us-central1)): redacted
Function URL (functionName(us-central1)): redacted
Function URL (functionName(us-central1)): redacted
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1

This is a small portion of the --debug log that printed for each failed function, but most of it did not seem useful:

  "original": {
    "name": "FirebaseError",
    "children": [],
    "exit": 1,
    "message": "Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted],
    "status": 3,
    "code": 3
  }

[trevor-rex]

Hello, I just wanted to check in on this and see if any devs have any idea what may be causing it? It is currently blocking us from having a consistent CI/CD pipeline, since even manually breaking up and batching the deployments isn't a valid workaround. Thank you!

[hunterua]

Same issue there.... have 20-30 functions and always have deployment error on random functions.... same error
Build failed with status: EXPIRED and message: An unexpected error occurred.

[Nushio]

Saw #5969 and thought it might be related to this,

Line 368 of extensionsApi.ts states masterTimeout: 600000 (and several other operations do, too).

I tried changing this locally for my firebase-tools, but i'm still getting the same error, Expired..., when trying to deploy ~40 gen2 functions.

[SikandAlex]

Same issue. Limited (< 10) number of Python 2nd Gen Firebase functions. Very frustrating and limiting the ability to productionize.

I think it happened when I added a 5th or 6th function. My functions have a large number of dependencies so not sure if that affects the issue. I haven't investigated the size in detail but am actively working to minimize the size.

Additionally, I am using both JS and Python functions with the codebase configuration set to different values. I am in the process of migrating the only JS function over to Python. Despite my best efforts, I couldn't get CORS to work on the 2nd gen Node.js functions so I couldn't get my local front-end to connect to my deployed backend (non-emulated) so moving to Python which I have had success with (but this is an issue for a different post).

I can also confirm that all of the functions are updated on each deployment instead of only those that have changes. This seems to be the current behavior of using the codebase configuration as mentioned in a linked PR / issue. Once I remove the need for codebase I will update this post.

Edit: No change after migration to using no codebase parameter and only Python functions. Considering moving to Terraform based deployments.

[hendrickson-tyler]

Also been dealing with this issue. Ever since I updated my 30 or so cloud functions to v2, I've been noticing errors when deploying some functions (the problematic ones change every time):

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].
✔  functions[payment-deleteOldPayments(us-central1)] Successful update operation.
✔  functions[customer-deleteJobsOnCustomerDelete(us-central1)] Successful update operation.
✔  functions[hourEntry-sendHourEntryReminders(us-central1)] Successful update operation.
✔  functions[calendar-addEvent(us-central1)] Successful update operation.
Functions deploy had errors with the following functions:
        calendar-getEvents(us-central1)
        job-deletePaymentsOnJobDelete(us-central1)
        payment-deletePaymentCheckPhoto(us-central1)
        subcontractor-onSubcontractorDelete(us-central1)
        user-deleteTasksOnUserDelete(us-central1)
Function URL (user-registerDeviceToken(us-central1)): [redacted]
Function URL (user-createNewUser(us-central1)): [redacted]
Function URL (calendar-getEventsToday(us-central1)): [redacted]
Function URL (calendar-addEvent(us-central1)): [redacted]
Function URL (calendar-updateEvent(us-central1)): [redacted]
Function URL (calendar-deleteEvent(us-central1)): [redacted]
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function calendar-getEvents in region us-central1
- Error Failed to update function payment-deletePaymentCheckPhoto in region us-central1
- Error Failed to update function user-deleteTasksOnUserDelete in region us-central1
- Error Failed to update function subcontractor-onSubcontractorDelete in region us-central1
- Error Failed to update function job-deletePaymentsOnJobDelete in region us-central1
Warning: run-commands command "firebase deploy --config firebase.json" exited with non-zero status code

Even worse, when these functions fail to deploy, subsequent deploys tell me that there are "no changes detected". There definitely are since those changes never made it to my function to begin with:

=== Deploying to 'crown-finish-development'...
i  deploying storage, firestore, functions, hosting
i  firebase.storage: checking apps/back-end/storage.rules for compilation errors...
✔  firebase.storage: rules file apps/back-end/storage.rules compiled successfully
i  firestore: reading indexes from apps/back-end/firestore.indexes.json...
i  cloud.firestore: checking apps/back-end/firestore.rules for compilation errors...
✔  cloud.firestore: rules file apps/back-end/firestore.rules compiled successfully
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
i  functions: Loading and analyzing source code for codebase default to determine what to deploy
Serving at port 8191
shutdown requested via /__/quitquitquit
i  functions: preparing dist/apps/back-end directory for uploading...
i  functions: packaged /Users/tylerhendrickson/Developer/soluna-studios/crown-finish/dist/apps/back-end (91.11 KB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
i  storage: latest version of apps/back-end/storage.rules already up to date, skipping upload...
i  firestore: deploying indexes...
✔  firestore: deployed indexes in apps/back-end/firestore.indexes.json successfully for (default) database
i  firestore: latest version of apps/back-end/firestore.rules already up to date, skipping upload...
i  hosting[crown-finish-development]: beginning deploy...
i  hosting[crown-finish-development]: found 50 files in dist/apps/web
i  hosting: upload complete
✔  hosting[crown-finish-development]: file upload complete
✔  storage: released rules apps/back-end/storage.rules to firebase.storage
✔  firestore: released rules apps/back-end/firestore.rules to cloud.firestore
i  functions: Skipping the deploy of unchanged functions with experimental support for skipdeployingnoopfunctions
✔  functions[user-deleteTasksOnUserDelete(us-central1)] Skipped (No changes detected)
✔  functions[user-deleteHourEntriesOnUserDelete(us-central1)] Skipped (No changes detected)
✔  functions[user-registerDeviceToken(us-central1)] Skipped (No changes detected)
✔  functions[user-createNewUser(us-central1)] Skipped (No changes detected)
✔  functions[job-deletePaymentsOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteReceiptsOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteJobPhotosOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteHourEntriesOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteTasksOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[hourEntry-sendHourEntryReminders(us-central1)] Skipped (No changes detected)
✔  functions[task-sendNotificationOnAssignment(us-central1)] Skipped (No changes detected)
✔  functions[task-sendTaskReminders(us-central1)] Skipped (No changes detected)
✔  functions[payment-deletePaymentCheckPhoto(us-central1)] Skipped (No changes detected)
✔  functions[payment-deleteOldPayments(us-central1)] Skipped (No changes detected)
✔  functions[receipt-deleteReceipt(us-central1)] Skipped (No changes detected)
✔  functions[receipt-deleteOldReceipts(us-central1)] Skipped (No changes detected)
✔  functions[estimate-deleteJobPhotosOnEstimateDelete(us-central1)] Skipped (No changes detected)
✔  functions[estimate-sendEstimateReminders(us-central1)] Skipped (No changes detected)
✔  functions[jobPhoto-deleteJobPhoto(us-central1)] Skipped (No changes detected)
✔  functions[jobPhoto-deleteOldJobPhotos(us-central1)] Skipped (No changes detected)
✔  functions[customer-updateJobsOnCustomerRename(us-central1)] Skipped (No changes detected)
✔  functions[customer-updateEstimatesOnCustomerRename(us-central1)] Skipped (No changes detected)
✔  functions[customer-deleteJobsOnCustomerDelete(us-central1)] Skipped (No changes detected)
✔  functions[customer-deleteEstimatesOnCustomerDelete(us-central1)] Skipped (No changes detected)
✔  functions[subcontractor-onSubcontractorDelete(us-central1)] Skipped (No changes detected)
✔  functions[calendar-getEvents(us-central1)] Skipped (No changes detected)
✔  functions[calendar-getEventsToday(us-central1)] Skipped (No changes detected)
✔  functions[calendar-addEvent(us-central1)] Skipped (No changes detected)
✔  functions[calendar-updateEvent(us-central1)] Skipped (No changes detected)
✔  functions[calendar-deleteEvent(us-central1)] Skipped (No changes detected)
✔  functions[expiration-recordCreationTime(us-central1)] Skipped (No changes detected)
i  functions: cleaning up build files...
i  hosting[crown-finish-development]: finalizing version...
✔  hosting[crown-finish-development]: version finalized
i  hosting[crown-finish-development]: releasing new version...
✔  hosting[crown-finish-development]: release complete
✔  Deploy complete!

Now I can't trust my CI/CD to deploy reliably. Some changes make it while others don't until I deploy next time with new changes. This is a huge problem. Can we please have some movement on this?

[hunterua]

Any updates on this? Any estimations? 😳