FCM Batch sending deprecated but no migration path

[19lmyers]

• Library version: 9.1.1
• Firebase Product: Messaging

I received an email from Firebase about the deprecation of legacy FCM APIs. Notably, the sendAll() method that employs batch sending has been deprecated, and I saw the Node Admin SDK has a new sendEach() method, but the Java SDK does not provide this method.

How should I approach migrating from the sendAll() method?

[AndrWeisR]

Ditto. I have firebase-admin-9.1.1.jar which only exposes these methods below. I've seen mention that these new methods are in firebase-admin 9.2.0, but that doesn't seem to be available anywhere yet.

[xak2000]

I see sendEach* methods are already implemented in the master.

The question is: when 9.2.0 will be released?

[adhirajsinghchauhan]

The question is: when 9.2.0 will be released?

@xak2000 v9.2.0 was released just a day after your comment.

---

But I wonder if it'd be feasible to walk back the decision to discontinue the batch send API, or at least move it into the Blaze plan? For people who want a fast, fire-and-forget delivery system, sendEach does not fit the bill. Both theoretically and in real-world tests, sendAll far outperforms its replacement.

Quick tests with 2000 messages (each to a different topic, but with the same config otherwise) on a few of my own servers. No code changes apart from calling a different function, no retry or delay mechanisms in between, apart from what the library itself defaults to:

 sendAll: 5-10s
sendEach: 1-5m

The overhead of 2000 HTTP calls instead of just 4 has obvious effects. This only becomes worse as the number of messages, meant to be sent approximately "together", increases.

[xak2000]

In the FAQ there is a dedicated topic about sending message to multiple tokens in one request.

Q. Does the HTTP v1 API support sending messages to multiple tokens in one request??
A. No. This feature, called "multicast" in legacy HTTP APIs, is not supported by the HTTP v1 API.

For use cases where end-to-end latency is critical, or where total fanout size is small (fewer than 1 million), Google recommends the HTTP V1 API. The HTTP V1 API over HTTP/2 can achieve up to ~1/3 of total message throughput rate compared to legacy multicast when sending on a single connection.

For use cases where throughput and egress bandwidth are prioritized or where total fanout size is large (greater than 1 million), Google recommends topic messaging. While topic messaging requires a one-time action to subscribe recipients to a topic, it offers up to a 10,000 QPS per project fanout rate without a maximum limit on topic size.

The question is: does firebase-admin-java use HTTP V1 API over HTTP/2? And, if yes, how to make SDK to use it? Probably, only Async variants of methods will use HTTP/2 multiplexing, right? Checked the code: both async and non-async variants call each request concurrently. The only difference is in handling the result. So, the only question is: does it use HTTP/2 multiplexing or each concurrent HTTP V1 API request creates a TCP connection?

[emindeniz99]

In the FAQ there is a dedicated topic about sending message to multiple tokens in one request.

Q. Does the HTTP v1 API support sending messages to multiple tokens in one request??
A. No. This feature, called "multicast" in legacy HTTP APIs, is not supported by the HTTP v1 API.
For use cases where end-to-end latency is critical, or where total fanout size is small (fewer than 1 million), Google recommends the HTTP V1 API. The HTTP V1 API over HTTP/2 can achieve up to ~1/3 of total message throughput rate compared to legacy multicast when sending on a single connection.
For use cases where throughput and egress bandwidth are prioritized or where total fanout size is large (greater than 1 million), Google recommends topic messaging. While topic messaging requires a one-time action to subscribe recipients to a topic, it offers up to a 10,000 QPS per project fanout rate without a maximum limit on topic size.

The question is: does firebase-admin-java use HTTP V1 API over HTTP/2? And, if yes, how to make SDK to use it? Probably, only Async variants of methods will use HTTP/2 multiplexing, right? Checked the code: both async and non-async variants call each request concurrently. The only difference is in handling the result. So, the only question is: does it use HTTP/2 multiplexing or each concurrent HTTP V1 API request creates a TCP connection?

Hello, i think it creates to many connection instead of using http2.
We faced with the issue at our system.
#849

[LucasJC]

Hey guys! I had the same experience using sendEach from previous comments.
Are there any plans for improving this new alternative?

[meor9zcu12]

Hi, I encountered the same issue. I plan to migrate fromsendMulticast to sendEachForMulticast, but the performance of sendEachForMulticast is very bad.

In my testing, to send 20,000 tokens
sendMulticast (sendAll API) : 4 seconds
sendEachForMulticast (sendEach API): 40 seconds

sendEach / sendEachForMulticast is useless for me until it is confirmed officially that supports http2 and have to restricted to http2 only. Otherwise high frequency of opening connection may make system crash.

---

In addition, I can't find any information about google-http-client:1.43.1 that does support http2

[AntQwanlity]

Found this : https://eladnava.com/send-multicast-notifications-using-node-js-http-2-and-the-fcm-http-v1-api/
Refers to this package : https://github.com/eladnava/fcm-v1-http2/tree/main
Might be useful for some.

However, I believe the Admin SDK should handle this under the hood. Having to manually add an HTTP/2 layer above the SDK is terrible DX. Might as well use OneSignal.

[emindeniz99]

The main issue is caused by google-http-client that does not support http2. default transport interface (NetHttpTransport) is JDK1.1 http connection.
I have tried different http transport layers for fcm library such as netty-reactor, java11-httpclient, springwebflux
googleapis/google-http-java-client@main...emindeniz99:google-http-java-client:main
https://github.com/emindeniz99/google-http-java-client

note: java 11 http client is not good for http2 requests, you should create multiple client for same target host in order to send multiple requests, it creates only one connection per host. I experienced the issue :D so I switched to netty-reactor
All of my tries are fast solutions to send http2 requests, can require improvements.

public FirebaseApp firebaseApp() throws IOException {
  GoogleCredentials credential = GoogleCredentials.fromStream(new FileInputStream(credentialsPath));

  ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat("Nio-firebase-http-client-%d").build();

  ConnectionProvider connectionProvider = ConnectionProvider.builder("myConnectionPool")
    .allocationStrategy(Http2AllocationStrategy.builder()
      .maxConnections(8)
      .minConnections(4)
      .maxConcurrentStreams(100) // google api responds http2 connection with 100 as max concurrent stream header
      .build())
    .pendingAcquireMaxCount(100_000)
    .build();

  //        EventLoopGroup bossGroup = new NioEventLoopGroup(2, namedThreadFactory);
  var client = HttpClient.create(connectionProvider

    )
    .protocol(HttpProtocol.H2)
  //                               .runOn(bossGroup)
  ;

  FirebaseOptions options =
    FirebaseOptions.builder().setCredentials(credential)
    .setHttpTransport(new ReactorNettyHttpTransport(client))
    .build();
  return FirebaseApp.initializeApp(options);
}

This is better, it does not open many sockets, etc. HTTP2 is used. It solves our main problem.
But the fcm library creates a thread for each request, for example sending 6 batch requests with each batch consisting of 500 device tokens. 6*500= 3k threads will be created.
As I know, each thread requires 1 MB memory, this is another problem.

Overriding the httptransport (FirebaseOptions...setHttpTransport) does not solve batch request issues completely. I do not want to change Firebase library, it requires a lot of effort and also I am not sure which approach is best for creating reactive or async library (CompletableFuture, Mono ...), and I do not have enough time, we are waiting for an official solution :D

#849

[sntg-p]

I was wondering if it makes sense to create a topic, make all your target devices subscribe to it and then send your message to that topic. Would that be an acceptable strategy?