Fix socket not reconnecting

[nduitz]

This handles error when the server closes connections.
The second commit implements a check for a open connection but it is not very resilient yet.
If the reconnect fails, the notification get's lost, either there needs to be some recovery mechanism or the response handler needs to be called.

#299

[nduitz]

Converted this to draft for now, I am still evaluating what else can be done so notifications won't get lost

[nduitz]

This got way bigger then expected but I think this is pretty solid now.
Would be glad if this got reviewed :)

This now handles mint errors while trying to push/reconnect to the different services.
If the push does not work, instead of throwing it away it get's requeued as a retry.
The number of retries can be configured, they use exponential backoffs between each retry (maybe should be configuriable too?)

I extracted a bit of adapter logic that applies to both apns + fcm to a helper. the ADM adapter does not use very much of it because it has a different behaviour

[Neylix]

Hey, thank's for the fix !

Just a remark on this line, since BackoffWorker is a named GenServer, we cannot start 2 of it. But as we can start multiple Dispatcher (one for APNS and one for FCM for example) the application cannot start :

** (Mix) Could not start application app_web: Veezu.AppWeb.Application.start(:normal, []) returned an error: shutdown: failed to start child: Veezu.Messaging.APNS
    ** (EXIT) shutdown: failed to start child: Pigeon.BackoffWorker
        ** (EXIT) already started: #PID<0.625.0>

Has there is no need to have one BackoffWorker per dispatcher, the fix is to start the BackoffWorker in the main application supervisor than per dispatcher

[mbarekTrismegistus]

hi, im using your fork in my project hoping it will fix the problem but i still get this:
** (MatchError) no match of right hand side value: {:error, %Mint.HTTP2{transport: Mint.Core.Transport.SSL, socket: {:sslsocket, {:gen_tcp, #Port<0.250>, :tls_connection, :undefined}, [#PID<0.3386.0>, #PID<0.3385.0>]}, mode: :active, hostname: "fcm.googleapis.com", port: 443, scheme: "https", authority: "fcm.googleapis.com", state: :closed, buffer: "", send_window_size: 1048576, receive_window_size: 16777216, receive_window_remaining: 16777216, receive_window_update_threshold: 160000, encode_table: %HPAX.Table{protocol_max_table_size: 4096, max_table_size: 4096, huffman_encoding: :never, entries: [], size: 0, length: 0, pending_minimum_resize: nil}, decode_table: %HPAX.Table{protocol_max_table_size: 4096, max_table_size: 4096, huffman_encoding: :never, entries: [], size: 0, length: 0, pending_minimum_resize: nil}, ping_queue: {[], []}, client_settings_queue: {[], []}, next_stream_id: 3, streams: %{}, open_client_stream_count: 0, open_server_stream_count: 0, reserved_server_stream_count: 0, ref_to_stream_id: %{}, server_settings: %{max_frame_size: 16384, initial_window_size: 1048576, max_concurrent_streams: 100, max_header_list_size: 65536, enable_push: true, enable_connect_protocol: false}, client_settings: %{max_frame_size: 16777215, initial_window_size: 2147483647, max_concurrent_streams: 100, max_header_list_size: 262144, enable_push: true}, headers_being_processed: nil, proxy_headers: [], private: %{}, log: false}, %Mint.HTTPError{reason: :closed, module: Mint.HTTP2}}

can i get an idea on why this happen ?