The Control Tower Behind ClickHouse

Server.cpp as the control tower

The report makes it clear that Server.cpp does not execute queries. Instead, it wires together configuration, caches, thread pools, network listeners, ZooKeeper/Keeper, metrics, reload callbacks, and shutdown logic. It’s an airport control tower: it never flies a plane, but one misordered step can bring the system down.

ClickHouse Server Entry (simplified)

repo root
├─ programs/
│  └─ server/
│     └─ Server.cpp   <-- this file
│
└─ src/ (core subsystems)
   ├─ Common/       (MemoryTracker, DNSResolver, ...)
   ├─ Interpreters/ (executeQuery, Context, ...)
   ├─ Storages/     (MergeTree, system tables, ...)
   ├─ Databases/    (database engines)
   └─ Server/       (HTTP handlers, TCP handlers, ...)

mainEntryClickHouseServer
  └─ DB::Server app
      └─ Server::main
          ├─ sanity checks & OS tuning
          ├─ context, caches, thread pools
          ├─ metadata & dictionaries
          ├─ protocol servers
          ├─ async metrics & config reload
          └─ graceful shutdown

The rest of the file is surprisingly coherent once you look at it through a lifecycle lens:

A complex server becomes understandable and evolvable when you treat it as a lifecycle: explicit phases of startup, live reconfiguration, and shutdown, each with clear responsibilities and invariants.

The following sections walk this lifecycle: how Server.cpp composes protocol stacks, reloads configuration safely, protects startup with checks and scripts, and bakes in operational guardrails. Each pattern is worth stealing for any serious server.

Protocols as pluggable stacks

The first place you see lifecycle‑oriented thinking is in how ClickHouse handles network protocols. Instead of hard‑coding “HTTP here, TCP there”, Server.cpp treats protocols as composable stacks configured at runtime: PROXY → TLS → HTTP, or TCP → MySQL, and so on. Under the hood this is a mix of the Adapter pattern (wrapping one interface into another) and a configuration‑driven Strategy (choosing behavior at runtime).

Building protocol stacks from config

The heart of this idea is Server::buildProtocolStackFromConfig. It reads sections and turns them into a chain of factories:

std::unique_ptr Server::buildProtocolStackFromConfig(
    const Poco::Util::AbstractConfiguration & config,
    const ServerSettings & server_settings,
    const std::string & protocol,
    Poco::Net::HTTPServerParams::Ptr http_params,
    AsynchronousMetrics & async_metrics,
    bool & is_secure)
{
    auto create_factory = [&](const std::string & type, const std::string & conf_name)
    {
        if (type == "tcp")
            return TCPServerConnectionFactory::Ptr(
                new TCPHandlerFactory(*this, false, false, ...));
        if (type == "tls")
            return TCPServerConnectionFactory::Ptr(new TLSHandlerFactory(*this, conf_name));
        if (type == "proxy1")
            return TCPServerConnectionFactory::Ptr(new ProxyV1HandlerFactory(*this, conf_name));
        if (type == "mysql")
            return TCPServerConnectionFactory::Ptr(new MySQLHandlerFactory(*this, ...));
        if (type == "http")
            return TCPServerConnectionFactory::Ptr(
                new HTTPServerConnectionFactory(
                    httpContext(), http_params,
                    createHandlerFactory(*this, config, async_metrics,
                                         "HTTPHandler-factory", handlers_config_key),
                    ...));
        // ...prometheus, interserver, postgres
    };

    std::string conf_name = "protocols." + protocol;
    std::string prefix = conf_name + ".";
    std::unordered_set visited {conf_name};

    auto stack = std::make_unique(*this, conf_name);

    while (true)
    {
        if (config.has(prefix + "type"))
        {
            std::string type = config.getString(prefix + "type");
            if (type == "tls")
                is_secure = true;
            stack->append(create_factory(type, conf_name));
        }

        if (!config.has(prefix + "impl"))
            break;

        conf_name = "protocols." + config.getString(prefix + "impl");
        prefix = conf_name + ".";

        if (!visited.insert(conf_name).second)
            throw Exception(ErrorCodes::INVALID_CONFIG_PARAMETER,
                "Protocol '{}' configuration contains a loop on '{}'", protocol, conf_name);
    }

    return stack;
}

A named protocol (for example, my_http) becomes a chain of layers by following type and optional impl links. The explicit loop‑detection prevents A → B → A cycles from turning into mysterious hangs.

Creating and restarting endpoints without duplication

Once stacks exist, Server.cpp uses a narrow set of helpers to manage endpoints throughout the lifecycle. The createServer helper encapsulates port binding, logging, and “best effort” vs “hard fail” behavior:

void Server::createServer(
    Poco::Util::AbstractConfiguration & config,
    const std::string & listen_host,
    const char * port_name,
    bool listen_try,
    bool start_server,
    std::vector & servers,
    CreateServerFunc && func) const
{
    if (config.getString(port_name, "").empty())
        return; // no port configured

    for (const auto & server : servers)
        if (!server.isStopping() &&
            server.getListenHost() == listen_host &&
            server.getPortName() == port_name)
            return; // already have one for this host+port

    auto port = config.getInt(port_name);
    try
    {
        servers.push_back(func(static_cast(port)));
        if (start_server)
        {
            servers.back().start();
            LOG_INFO(&logger(), "Listening for {}", servers.back().getDescription());
        }
        global_context->registerServerPort(port_name, static_cast(port));
    }
    catch (const Poco::Exception &)
    {
        if (listen_try)
        {
            LOG_WARNING(&logger(), "Listen [{}]:{} failed: {} ...",
                        listen_host, port, getCurrentExceptionMessage(false));
        }
        else
        {
            throw Exception(ErrorCodes::NETWORK_ERROR,
                "Listen [{}]:{} failed: {}",
                listen_host, port, getCurrentExceptionMessage(false));
        }
    }
}

The lifecycle benefit:

• Protocol composition is independent from socket binding.
• Socket binding is independent from starting and registering servers.
• Error policy (listen_try vs hard failure) lives in one place.

The same helpers are reused on startup and during config reload. That reuse is what makes hot‑reload practical instead of a bolted‑on afterthought.

Safe live reconfiguration

Once the process is live, the hardest part of the lifecycle is reconfiguration. Changing memory limits, cache sizes, and ports without downtime is surgery on a running system. ClickHouse does this through a ConfigReloader that calls a single (large) lambda to apply configuration to shared components.

Reload as recomputation, not incremental tweaks

The report flags the reload lambda as complex, but it also highlights a crucial idea: on each reload, recompute derived values from first principles instead of nudging old state. Here’s a condensed excerpt:

auto main_config_reloader = std::make_unique(
    config_path,
    extra_paths,
    server_settings[ServerSetting::path],
    std::move(main_config_zk_node_cache),
    main_config_zk_changed_event,
    [&](ConfigurationPtr loaded_config, bool initial_loading)
    {
        config().replace("default", loaded_config, PRIO_DEFAULT, true);

        ServerSettings new_server_settings;
        new_server_settings.loadSettingsFromConfig(config());

        size_t max_server_memory_usage =
            new_server_settings[ServerSetting::max_server_memory_usage];
        const double ratio =
            new_server_settings[ServerSetting::max_server_memory_usage_to_ram_ratio];
        const size_t current_ram = getMemoryAmount();
        const size_t default_limit =
            static_cast(static_cast(current_ram) * ratio);

        if (max_server_memory_usage == 0 || max_server_memory_usage > default_limit)
            max_server_memory_usage = default_limit;

        total_memory_tracker.setHardLimit(max_server_memory_usage);

        const size_t max_cache_size_in_bytes = static_cast(
            static_cast(current_ram) *
            new_server_settings[ServerSetting::cache_size_to_ram_max_ratio]);

        global_context->updateUncompressedCacheConfiguration(
            config(), max_cache_size_in_bytes);
        global_context->updateMarkCacheConfiguration(
            config(), max_cache_size_in_bytes);
        // ...other caches and limits...

        if (global_context->isServerCompletelyStarted())
        {
            std::lock_guard lock(servers_lock);
            updateServers(config(), new_server_settings,
                          server_pool, *async_metrics,
                          servers, servers_to_start_before_tables);
        }

        latest_config = loaded_config;
    });

A few lifecycle‑critical properties:

• Limits derive from current RAM: both max_server_memory_usage and cache envelopes are recomputed from current physical memory and ratios. If the container’s memory limit changes, the next reload adjusts caps accordingly.
• Ordering is deliberate: memory trackers and caches are updated first; only then are protocol servers updated under servers_lock. This minimizes contention and avoids inconsistent state.
• Initial load is special‑cased: on first load, the callback avoids work that requires the server to be “completely started”, preventing weird half‑initialized states.

Hot‑reloading servers by replacement

Protocol servers themselves are hot‑reloaded via Server::updateServers. Instead of mutating servers in place, ClickHouse stops and replaces them when configuration changes:

void Server::updateServers(
    Poco::Util::AbstractConfiguration & config,
    const ServerSettings & server_settings,
    Poco::ThreadPool & server_pool,
    AsynchronousMetrics & async_metrics,
    std::vector & servers,
    std::vector & servers_to_start_before_tables)
{
    LoggerRawPtr log = &logger();

    const auto listen_hosts = getListenHosts(config);
    const auto interserver_listen_hosts = getInterserverListenHosts(config);
    const auto listen_try = getListenTry(config, server_settings);

    auto check_server = [&log](const char prefix[], auto & server)
    {
        if (!server.isStopping())
            return false;
        size_t current_connections = server.currentConnections();
        LOG_DEBUG(log,
            "Server {}{}: {} ({} connections)",
            server.getDescription(), prefix,
            !current_connections ? "finished" : "waiting",
            current_connections);
        return !current_connections;
    };

    std::erase_if(servers,
        std::bind_front(check_server,
                        " (from one of previous reload)"));

    Poco::Util::AbstractConfiguration & previous_config =
        latest_config ? *latest_config : config;

    std::vector all_servers;
    all_servers.reserve(servers.size() +
                        servers_to_start_before_tables.size());
    for (auto & s : servers) all_servers.push_back(&s);
    for (auto & s : servers_to_start_before_tables)
        all_servers.push_back(&s);

    for (auto * server : all_servers)
    {
        if (server->supportsRuntimeReconfiguration() &&
            !server->isStopping())
        {
            std::string port_name = server->getPortName();
            bool has_host = ...;      // host still configured?
            bool force_restart = ...; // handlers or port changed?

            if (!has_host || !has_port || port_changed || force_restart)
            {
                server->stop();
                LOG_INFO(log,
                    "Stopped listening for {}",
                    server->getDescription());
            }
        }
    }

    createServers(config, server_settings, listen_hosts,
                  listen_try, server_pool, async_metrics,
                  servers, true);
    createInterserverServers(config, server_settings,
                             interserver_listen_hosts,
                             listen_try, server_pool,
                             async_metrics,
                             servers_to_start_before_tables,
                             true);

    std::erase_if(servers, std::bind_front(check_server, ""));
    std::erase_if(servers_to_start_before_tables,
                  std::bind_front(check_server, ""));
}

The lifecycle principle is straightforward:

• Once constructed, a ProtocolServerAdapter is treated as immutable.
• Configuration changes cause stop‑and‑replace, not in‑place mutation.
• Cleanup of drained servers is centralized via check_server.

Startup, checks, and automation

The lifecycle begins before ClickHouse reads its main config. The thin entrypoint sets up a watchdog if needed and delegates all work to DB::Server. After that, Server::main runs environment checks and optional startup scripts before marking the server as ready.

Entry point and watchdog

The top‑level entry is intentionally simple:

int mainEntryClickHouseServer(int argc, char ** argv)
{
    DB::Server app;

    if (argc > 0)
    {
        const char * env_watchdog = getenv("CLICKHOUSE_WATCHDOG_ENABLE");
        if (env_watchdog)
        {
            if (0 == strcmp(env_watchdog, "1"))
                app.shouldSetupWatchdog(argv[0]);
        }
        else if (!isatty(STDIN_FILENO) &&
                 !isatty(STDOUT_FILENO) &&
                 !isatty(STDERR_FILENO))
        {
            app.shouldSetupWatchdog(argv[0]);
        }
    }

    try
    {
        return app.run(argc, argv);
    }
    catch (...)
    {
        std::cerr << DB::getCurrentExceptionMessage(true)
                  << "\n";
        auto code = DB::getCurrentExceptionCode();
        return static_cast(code) ? code : 1;
    }
}

Two things matter here for the lifecycle:

• Environment‑driven behavior: watchdog setup is decided using CLICKHOUSE_WATCHDOG_ENABLE and TTY checks, because configuration is not yet available.
• Top‑level exception safety: any uncaught exception is rendered as a message and exit code instead of a silent crash.

Sanity checks as structured warnings

After configuration and context are initialized, sanityChecks inspects OS‑level settings and environment quality. Instead of aborting on suboptimal setups, it records structured warnings in the context:

void sanityChecks(Server & server,
                  const ServerSettings & server_settings)
{
    std::string data_path = getCanonicalPath(
        String(server_settings[ServerSetting::path]),
        server.getOriginalWorkingDirectory());

#if defined(OS_LINUX)
    try
    {
        const char * filename =
            "/sys/devices/system/clocksource/clocksource0/current_clocksource";
        if (!fast_clock_sources.contains(readLine(filename)))
            server.context()->addOrUpdateWarningMessage(
                Context::WarningType::LINUX_FAST_CLOCK_SOURCE_NOT_USED,
                PreformattedMessage::create(
                    "Linux is not using a fast clock source. Check {}",
                    filename));
    }
    catch (const std::exception &) {}

    try
    {
        const char * filename = "/proc/sys/vm/overcommit_memory";
        if (readNumber(filename) == 2)
            server.context()->addOrUpdateWarningMessage(
                Context::WarningType::LINUX_MEMORY_OVERCOMMIT_DISABLED,
                PreformattedMessage::create(
                    "Linux memory overcommit is disabled. Check {}",
                    filename));
    }
    catch (const std::exception &) {}
    // ... hugepages, pid_max, threads-max, mdraid, disk space, etc.
#endif

    try
    {
        if (getAvailableMemoryAmount() < (2l << 30))
            server.context()->addOrUpdateWarningMessage(
                Context::WarningType::AVAILABLE_MEMORY_TOO_LOW,
                PreformattedMessage::create(
                    "Available memory at server startup is too low (2GiB)."));
    }
    catch (const std::exception &) {}

    // ... other checks for disk space, log paths, replication settings
}

These checks fit neatly into the lifecycle model:

• They run once during startup, when environment is stable.
• They record machine‑readable warnings into context, not just ad‑hoc log lines.
• Operators can query and alert on them later via system tables.

Startup scripts with guardrails

The last startup phase before serving traffic is optional automation through loadStartupScripts. Admins can configure SQL to run on startup, gated by conditions and executed as dedicated users. That’s powerful enough to change data and schema, so the implementation adds several guardrails:

void loadStartupScripts(const Poco::Util::AbstractConfiguration & config,
                        const ServerSettings & server_settings,
                        ContextMutablePtr context,
                        Poco::Logger * log)
{
    try
    {
        Poco::Util::AbstractConfiguration::Keys keys;
        config.keys("startup_scripts", keys);

        std::vector skipped_startup_scripts;

        for (const auto & key : keys)
        {
            if (key == "throw_on_error")
                continue;
            std::string full_prefix =
                "startup_scripts." + key;

            auto user = config.getString(
                full_prefix + ".user", "");
            auto startup_context =
                Context::createCopy(context);
            if (!user.empty())
            {
                auto & access_control =
                    startup_context->getAccessControl();
                startup_context->setUser(
                    access_control.getID(user));
            }

            if (config.has(full_prefix + ".condition"))
            {
                auto condition = config.getString(
                    full_prefix + ".condition");
                // executeQuery(condition) and interpret result as boolean
                if (result != "1\n" && result != "true\n")
                {
                    if (result != "0\n" &&
                        result != "false\n")
                        skipped_startup_scripts.emplace_back(
                            full_prefix);
                    continue;
                }
            }

            auto query = config.getString(
                full_prefix + ".query");
            LOG_DEBUG(log,
                "Executing query `{}`", query);
            executeQuery(...);
        }

        if (!skipped_startup_scripts.empty())
        {
            context->addOrUpdateWarningMessage(...);
        }

        CurrentMetrics::set(
            CurrentMetrics::StartupScriptsExecutionState,
            StartupScriptsExecutionState::Success);
    }
    catch (...)
    {
        DimensionalMetrics::set(
            DimensionalMetrics::StartupScriptsFailureReason,
            {String(ErrorCodes::getName(
                getCurrentExceptionCode()))},
            1.0);

        CurrentMetrics::set(
            CurrentMetrics::StartupScriptsExecutionState,
            StartupScriptsExecutionState::Failure);
        tryLogCurrentException(
            log,
            "Failed to parse startup scripts file");
        if (server_settings[
            ServerSetting::startup_scripts_throw_on_error])
            throw Exception(
                ErrorCodes::STARTUP_SCRIPTS_ERROR,
                "Cannot finish startup_script successfully. "
                "Use startup_scripts.throw_on_error...");
    }
}

Lifecycle‑wise, startup scripts are their own phase with clear semantics:

• They run under explicitly configured users.
• They can be skipped based on condition queries.
• Their success/failure is tracked by metrics and can be made fatal via config.

That gives you automation without turning “run whatever on startup” into an unobservable risk.

Operational and scalability guardrails

With startup, reload, and shutdown wired up, the last question for the lifecycle is: how does this design behave under real load and failures? The report’s performance discussion and metric suggestions show how Server.cpp keeps itself observable and within safe operating bounds.

Hot and semi‑hot paths owned by Server.cpp

Query execution lives elsewhere, but Server.cpp still drives some hot or semi‑hot paths:

• Asynchronous metrics: a background AsynchronousMetrics thread periodically iterates over ProtocolServerAdapter instances (under servers_lock) to collect connection counts and thread counts.
• Config reload: infrequent but heavy, updating memory trackers, caches, throttlers, and servers in one pass.
• Memory worker: a MemoryWorker tunes RSS and page cache usage, influencing how the process interacts with the OS under pressure.

These are designed to be either infrequent or O(number of servers / caches), which is tiny relative to query volume, so the control tower doesn’t become a bottleneck.

Metrics that expose lifecycle health

The report proposes several metrics that are especially valuable when you view the process as a lifecycle. They’re worth adopting conceptually even outside ClickHouse:

Scalability limits enforced at the edges

Many scalability guardrails are applied in Server::main and the reload callback rather than deep inside subsystems:

• Process limits: the server attempts to raise RLIMIT_NOFILE and RLIMIT_NPROC and logs warnings when threads-max is too low (for example, below 30,000), instead of discovering these limits only under load.
• Memory envelope: max_server_memory_usage and merges_mutations_memory_usage_soft_limit are computed from RAM and ratios and then enforced via total_memory_tracker and related trackers.
• Cache envelopes: all cache sizes are compared against a RAM‑based max_cache_size_in_bytes; if configuration overshoots, sizes are lowered and the adjustment is logged.
• Concurrency control: the global concurrent_threads_soft_limit and a scheduler are configured centrally, giving one place to reason about CPU slot allocation.

Putting these rules at the lifecycle boundaries means most misconfigurations are corrected or at least surfaced early, instead of showing up as weird runtime failures.

What to steal for your own servers

Stepping back, Server.cpp shows how to keep a complex server evolvable by treating it as a clear lifecycle: startup, steady‑state, reload, and shutdown, each with explicit responsibilities, safety rails, and metrics. The file is large, but it reads like a flight plan, not a grab bag of hacks.

1. Make lifecycle phases explicit

The report suggests refactoring Server::main into functions such as initEnvironment, initCachesAndMemory, startMetadataAndServers, and shutdownServersAndResources. Even before you do that refactor, you can impose the discipline that every new feature belongs to a named phase and lives next to similar concerns.

If you can’t say which lifecycle phase a change belongs to, it’s probably leaking into the wrong part of the system.

2. Centralize endpoint and protocol management

ClickHouse pushes all socket creation and protocol wiring through a small set of helpers (buildProtocolStackFromConfig, createServer, createServers, updateServers) guarded by a shared lock. The report even recommends extracting this further into a dedicated ServerEndpoints helper.

For your own servers, resist the urge to sprinkle listen() calls across the codebase. One module should own “what do we listen on, and how do we change it at runtime?”.

3. Treat configuration as a pure function

Server.cpp repeatedly computes derived values from raw configuration and environment (memory limits from RAM, cache sizes from ratios, etc.), both at startup and on reload. The report explicitly recommends extracting these computations into reusable functions to avoid duplication.

That’s a good pattern elsewhere: define helpers like computeMemoryLimits(env, settings) and call them from every lifecycle phase that needs them. It makes behavior consistent and easier to test.

4. Prefer structured warnings and metrics over guesswork

From sanityChecks to startup scripts to reload failures, unusual situations become context warnings and metrics instead of only log lines. That’s what lets you drive dashboards and alerts from the control tower, rather than grepping logs at 3 a.m.

5. Make hot‑reload safe by constructing, then swapping

Protocol servers, caches, and many thread pools are treated as replaceable: you build a fully configured instance from config, stop the old one, insert the new one, and then clean up drained resources. In‑place mutation is avoided wherever possible.

Adopt the same mindset: design components so they can be constructed from configuration plus context and then atomically swapped into the running system.

The overarching lesson from ClickHouse’s Server.cpp is simple but strict: your main server file is not just an entrypoint, it is your control tower. If you make its lifecycle phases explicit, keep protocol and endpoint management centralized, recompute configuration consistently, and expose everything via structured warnings and metrics, you can keep scaling both the system and the team that works on it.