Add runtime setters for MULTI_POLL reactor count, dispatch, and request ceiling

[Matt711]

Adds setters for remote_io_num_reactors, remote_io_reactor_dispatch, and remote_io_max_concurrent_requests

• Need for cudf-polars Default cudf-polars to the kvikio MULTI_POLL backend NVIDIA/cudf#23839

[madsbk]

Overall looks good.

• update the stale sentence at runtime_settings.rst:82
• add missing docs for three newly user-facing settings

[madsbk]

I think it would be helpful if we could throw if the reactor pool has already been started?
Maybe add something like:

// multi_poll_reactor.hpp
static bool is_instantiated() noexcept;

// multi_poll_reactor.cpp
namespace { std::atomic<bool> _pool_instantiated{false}; }

bool MultiReactorPool::is_instantiated() noexcept
{
  return _pool_instantiated.load(std::memory_order_acquire);
}

MultiReactorPool::MultiReactorPool() : _dispatch{defaults::remote_io_reactor_dispatch()}
{
  ...
  _pool_instantiated.store(true, std::memory_order_release);  // end of ctor
}

[wence-]

nit: Like Mads, I think we should throw if this set call would not take effect.

[wence-]

Same here.

[wence-]

And here.

[wence-]

Yeah, I think especially because the getters will return this value (even if it was not the one used for setting up the multi-poll instance), we should throw. Otherwise inspecting the defaults later to report behaviour will be misleading.

[kingcrimsontianyu]

I agree with the suggestion. Just want to point out that for the BS thread pool used for local I/O and easy backend, we do support changing the number of threads midway at runtime, which would block the calling thread, drain the task queue, destroy the worker threads, and then recreate a new set of worker threads. So I think the phrasing "like the default thread pool" should be removed from the doc string here.

[madsbk]

This need a guard, I don't think it will work with -DKvikIO_REMOTE_SUPPORT=OFF ?

[madsbk]

Since we cannot reset the pool size, the defaults context manager is now broken:

with kvikio.defaults.set("remote_io_num_reactors", 8):
    remote_file.pread(...)

It fails when trying to reset remote_io_num_reactors on exit.