Gen 2 function deploys can exhaust regional CPU/memory because release concurrency is fixed at 40

[EthanSK]

[REQUIRED] Environment info

firebase-tools: 15.25.1. I also checked 15.28.2, the latest release at the time of filing, and the relevant concurrency remains unchanged.

Platform: Ubuntu (GitHub Actions), Node.js 22

[REQUIRED] Test case

A Firebase project with approximately 70 Gen 2 functions in one region. Many functions share a bundle, so a shared change causes most of them to require new revisions in one deploy.

The project's regional Cloud Run quota is 20 vCPU / 40 GiB.

[REQUIRED] Steps to reproduce

1. Change shared function code so many Gen 2 functions need an update.
2. Run:

firebase deploy --only functions --project <project> --non-interactive --force

3. Observe many Gen 2 update operations being released concurrently.

The Gen 2 update path uses functionExecutor, while the release executor has a fixed concurrency of 40:

• firebase-tools/src/deploy/functions/release/index.ts

  Lines 80 to 104 in ec32692

  	const fnsToUpdateSafe = await prompts.promptForUnsafeMigration(fnsToUpdate, options);
  	const safeEndpoints = new Set(fnsToUpdateSafe.map((eu) => eu.endpoint));
  	for (const changes of allRegionalChanges) {
  	changes.endpointsToUpdate = changes.endpointsToUpdate.filter((eu) =>
  	safeEndpoints.has(eu.endpoint),
  	);
  	}
  	const throttlerOptions = {
  	retries: 30,
  	backoff: 20000,
  	concurrency: 40,
  	maxBackoff: 100000,
  	};
  	// N.B. THIS IS TEMPORARY
  	// This will limit concurrent deploys of run functions to two while zip deploy capacity
  	// is low.
  	const runThrottlerOptions = {
  	...throttlerOptions,
  	concurrency: 2,
  	};
  	const projectNumber = options.projectNumber || (await getProjectNumber(context.projectId));

• firebase-tools/src/deploy/functions/release/fabricator.ts

  Lines 760 to 785 in ec32692

  	async updateV2Function(endpoint: backend.Endpoint, scraper: SourceTokenScraper): Promise<void> {
  	const storageSource = this.sources[endpoint.codebase!]?.storage;
  	if (!storageSource) {
  	logger.debug("Precondition failed. Cannot update a GCFv2 function without storage");
  	throw new Error("Precondition failed");
  	}
  	const apiFunction = gcfV2.functionFromEndpoint({ ...endpoint, source: { storageSource } });
  	// N.B. As of GCFv2 private preview the API chokes on any update call that
  	// includes the pub/sub topic even if that topic is unchanged.
  	// We know that the user hasn't changed the topic between deploys because
  	// of checkForInvalidChangeOfTrigger().
  	if (apiFunction.eventTrigger?.pubsubTopic) {
  	delete apiFunction.eventTrigger.pubsubTopic;
  	}
  	const resultFunction = await this.functionExecutor
  	.run(
  	async () => {
  	if (experiments.isEnabled("functionsv2deployoptimizations")) {
  	apiFunction.buildConfig.sourceToken = await scraper.getToken();
  	}
  	const op: { name: string } = await gcfV2.updateFunction(apiFunction);
  	return await poller.pollOperation<gcfV2.OutputCloudFunction>({
  	...gcfV2PollerOptions,
  	pollerName: `update-${endpoint.codebase}-${endpoint.region}-${endpoint.id}`,

firebase deploy --help does not expose a supported way to lower this concurrency.

[REQUIRED] Expected behavior

A bulk functions deploy should schedule Gen 2 updates so temporary revision health checks do not exhaust the project's regional CPU/memory quota.

At minimum, the CLI should expose a supported concurrency setting so CI can choose a value appropriate for the project's quota and function CPU/memory sizes. A quota-aware default would be even better.

[REQUIRED] Actual behavior

One deploy produced 56 errors like these, many within the same second:

Could not create or update Cloud Run service <service>, Container Healthcheck failed.
Quota exceeded for total allowable CPU per project per region.

The deployment failed even though the services themselves were healthy and deployable individually.

Workarounds tried:

• CPU-aware batches still failed because old/new revision overlap and deployment health checks consume regional quota.
• Fixed 15-minute waits made the deployment take more than 90 minutes and still eventually hit the memory quota.
• Pre-updating existing Cloud Run services with gcloud run services update --no-deploy-health-check allowed the same bulk Firebase deployment to complete, but this removes eager startup validation and cannot prepare a service that does not exist yet.

This is specifically about aggregate CPU/memory consumed by concurrent Gen 2 revision deployment health checks. It is separate from Cloud Functions API write-rate quota errors and from a function's configured maxInstances being individually too high.

Related older reports cover retries/rate limits or the CLI exit status, but not a supported limit for this Gen 2 release concurrency:

• Request: Automatically retry failed function deploys that fail due to "You have exceeded your deployment quota" #2606
• function deploy Quota Exceeded error not considered error #7949

[IzaakGough]

Hi @EthanSK, thanks for raising this. We will review as soon as possible

[IzaakGough]

Hi @EthanSK, thanks for the detailed write up.

It looks like the concurrency: 40 is project-wide rather than per region, and because each update waits on the shared build source token inside its queue slot, the slots fill up and then fire together, which explains the errors landing in the same second.

The update path retries Cloud Run RESOURCE_EXHAUSTED, but only on RPC code 8. If your healthcheck failure has a different code, nothing retries it. If you still have a run available, a --debug log would confirm that, since the raw operation error is logged there.

On next steps: the retry gap looks like a straightforward bug fix. Configurable concurrency seems like a good feature request, so I will raise that internally and keep you updated here.

[google-oss-bot]

Hey @EthanSK. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@EthanSK if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.