Skip to content

Add non-blocking DNS for the async API via c-ares event-loop integration#324

Draft
bjosv wants to merge 6 commits into
valkey-io:mainfrom
bjosv:use-cares-async
Draft

Add non-blocking DNS for the async API via c-ares event-loop integration#324
bjosv wants to merge 6 commits into
valkey-io:mainfrom
bjosv:use-cares-async

Conversation

@bjosv

@bjosv bjosv commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Even with c-ares enabled for timeout-bounded DNS, the async API still blocked the calling thread during DNS resolution. valkeyAsyncConnectWithOptions() resolved DNS synchronously before returning, using the same
blocking resolve path as the sync API. For applications using async I/O on a single thread, a slow DNS lookup stalls the entire event loop.

This PR builds on the c-ares sync DNS support (#323) and makes DNS fully non-blocking for the async API. When the connection is set up via the options path with a c-ares aware adapter (libevent), DNS resolution
is driven by the event loop, so no thread blocks during DNS.

How it works:

Async DNS is opt-in: it engages only when connecting via valkeyAsyncConnectWithOptions() with an adapter set in valkeyOptions.attach_fn, and only when that adapter implements the c-ares hooks. When c-ares is
enabled and such a connection is made:

  1. valkeyAsyncConnectWithOptions() defers the TCP connect (VALKEY_CONNECT_DEFERRED): the endpoint is stored, but no DNS lookup or socket fd is created yet.
  2. It then attaches the adapter (before initiating connect), so the adapter's hooks are registered first:
    • Libevent adapter (implements the c-ares hooks): starts a non-blocking DNS query via c-ares. The c-ares file descriptors and timers are registered with the event loop, so DNS completes as part of normal
      event processing without blocking.
    • Other adapters (libev, libuv, poll, etc.): do not implement the hooks, so valkeyAsyncConnectWithOptions() resolves DNS synchronously with a timeout and connects before returning. This still blocks briefly
      but is bounded by connect_timeout. These adapters can be upgraded to full async DNS by implementing the c-ares hooks.
  3. Once DNS resolves, a TCP socket is created and connect() is called (non-blocking, as usual). The event loop then handles connect completion.

The legacy path (valkeyAsyncConnect() followed by a separate adapter attach) never defers, and always uses the blocking, timeout-bounded resolve, unchanged.

The connect logic (source_addr binding, SO_REUSEADDR, MPTCP) is consolidated in valkeyTcpConnectNonBlock() to avoid duplication between the async DNS path and the synchronous fallback.

Design decisions:

  • Async DNS is deliberately opt-in via the options path. Any adapter without c-ares hooks, and the entire legacy connect-then-attach path, get blocking-with-timeout DNS. This keeps behavior predictable and
    requires no changes to existing adapters.
  • The blocking fallback is centralized in valkeyAsyncConnectWithOptions() (taken when the adapter did not start async DNS, i.e. dns_state == NULL), rather than a per-adapter macro. Adapters need no changes.
  • c-ares callbacks can fire synchronously (e.g. IP literals like 127.0.0.1). A done/failed flag pattern ensures cleanup happens safely outside the c-ares callback. An ARES_EDESTRUCTION guard in the callback
    prevents side effects when the channel is destroyed during context teardown.
  • While the connect is deferred there is no valid fd, so valkeyAsyncConnectWithOptions() and valkeyAsyncSetConnectCallback() skip write-interest registration; the write watch is added once the fd exists (after
    sync connect, or from the DNS-completion callback). The shared _EL_ADD_READ/_EL_ADD_WRITE macros are unchanged.
  • Async DNS is driven through the existing internal timer system (no new adapter timer hook).

Custom adapters:

Third-party adapters need no changes to remain correct: when c-ares is enabled, an adapter that does not implement the hooks gets a synchronous, timeout-bounded resolve performed by
valkeyAsyncConnectWithOptions() before the adapter is asked to operate on the fd. To get fully non-blocking DNS, an adapter implements the c-ares socket hooks (addCaresSocket/delCaresSocket) and calls
valkeyResolveAsyncStart() from its attach function, forwarding fd activity to valkeyResolveAsyncHandleEvent(). See the libevent adapter for a reference.

Testing:

  • New ct_async_dns integration test exercising both the libevent async path and the standalone connect flow.

bjosv added a commit that referenced this pull request Jun 24, 2026
`getaddrinfo()` has no timeout mechanism. When DNS is slow or
unresponsive, libvalkey hangs indefinitely with no way to cancel the
lookup.

This PR adds optional c-ares integration (`USE_CARES=1` /
`-DENABLE_CARES=ON`) that replaces `getaddrinfo()` with c-ares driven by
a `poll()` loop. DNS resolution respects `connect_timeout` (defaulting
to 5 seconds if unset), so lookups that previously hung indefinitely now
fail with a timeout error.

The PR consists of two commits: the first moves existing DNS logic into
`dns.[ch]`, and the second adds the c-ares integration.

**Notes:**
- The async API currently still blocks during DNS (now with a timeout).
A follow-up PR
  #324
will make DNS fully non-blocking for the async API via event-loop
integration.
- Without c-ares enabled, behavior is unchanged (plain `getaddrinfo`)
- IPv4/IPv6 preference flags and fallback behavior are preserved
- IPv6 literals (e.g. `::1`) are detected and handled correctly
- Targets Linux, macOS, FreeBSD (not Windows)

---------

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
@bjosv bjosv force-pushed the use-cares-async branch from 88bcf5c to 1e6a824 Compare June 24, 2026 07:56
bjosv added 5 commits July 3, 2026 13:53
Add attach_fn and attach_data to valkeyOptions, allowing the event-loop
adapter to be specified as part of the connection options. When set,
valkeyAsyncConnectWithOptions() attaches the adapter and registers the
fd with the event loop internally.

For TCP connections, connect is deferred past context initialization to
ensure the context is fully set up before the connect call. The adapter
receives a valid fd and works without modification.

This mirrors the existing cluster API pattern (valkeyClusterOptions) and
enables future work where the adapter drives non-blocking DNS resolution.

The legacy path (attaching the adapter separately after connect) remains
fully supported.

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
Add async_connect_callback and async_disconnect_callback to valkeyOptions,
allowing the connect and disconnect callbacks to be specified as part of the
connection options.

When set, valkeyAsyncConnectWithOptions() registers them on the context before
the connect is initiated, so no connect event is missed when the event loop is
already running at connect time. This is equivalent to calling
valkeyAsyncSetConnectCallback() / valkeyAsyncSetDisconnectCallback() after
connect, which remains fully supported.

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
Add src/timer.c with a pre-allocated pool of timers managed as a
sorted linked list. Timers can be scheduled, cancelled, and
processed one at a time (to allow callbacks to free the context).

This is the foundation for running multiple concurrent timers
(command timeout, connect timeout, cluster topology refresh, etc.)
through a single adapter scheduleTimer hook.

Callers that need periodic behavior reschedule in their callback.

Uses CLOCK_MONOTONIC on Unix and QueryPerformanceCounter on Windows.

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
Connect and command timeouts are now separate timers:
- connect_timer: scheduled during connect, cancelled on success
- command_timer: scheduled on first write when connected, tracks
  reply activity to detect unresponsive servers

valkeyAsyncHandleTimeout dispatches expired timers via
valkeyProcessTimers. No adapter changes required.

| Test                     | main   | PR     |
|--------------------------|--------|--------|
| 10000x PING pipelined    | 0.004s | 0.005s | (avg from 4 runs)
| 10000x INCRBY pipelined  | 0.004s | 0.004s | (avg from 4 runs)
| 10000x LRANGE pipelined  | 0.793s | 0.782s | (avg from 4 runs)

Note: changing command_timeout at runtime no longer affects an
already-running timer. The new value takes effect after the current
timer fires.

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
@bjosv bjosv force-pushed the use-cares-async branch 2 times, most recently from ee27b49 to 9cdafe2 Compare July 5, 2026 12:24
When c-ares is enabled and the connection is set up via the options
path (valkeyAsyncConnectWithOptions with attach_fn), DNS resolution is
driven by the event loop instead of blocking the calling thread.

For TCP, connect is deferred until the event adapter is attached. A
c-ares aware adapter (libevent) initiates async DNS from its attach
function, registering c-ares sockets and timers with the event loop.
On DNS completion, socket() + connect() proceed as normal and the fd
is registered for connect completion.

The async path is opt-in: it only engages through the options path with
an adapter that implements the c-ares hooks (addCaresSocket /
delCaresSocket). All other cases, the legacy connect-then-attach path
and adapters without the hooks, resolve DNS synchronously with the
existing timeout-bounded path.

Changes:
- Add addCaresSocket/delCaresSocket hooks and dns_state to
  valkeyAsyncContext
- Add the async DNS engine to dns.c (valkeyResolveAsyncStart /
  HandleEvent / Free), driven by the internal timer system
- Add valkeyTcpConnectNonBlock to net.c for connect after async resolve
- Attach the adapter before initiating connect in
  valkeyAsyncConnectWithOptions so async DNS can start on attach
- Implement the c-ares socket handling in the libevent adapter, with
  lazy fd registration since no fd exists until DNS completes

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
@bjosv bjosv force-pushed the use-cares-async branch from 9cdafe2 to 07b23ad Compare July 5, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant