Posted in

C++ AI-generated code that looks fast, but is actually slow

In this article we will cover the following cases:

  • std::end vs \n
  • branchless logic
  • preferring function objects over templates

std::endl vs \n

One of the most common examples of AI-generated C++ code that looks modern and professional, but is actually terrible for performance, is the overuse of std::endl. Large language models generate it constantly because it appears in countless tutorials, Stack Overflow answers, textbooks and introductory examples. It also looks more “explicit” and “system-level” than simply writing \n, which aligns with the kind of code patterns AI tends to associate with high-performance C++.

The problem is that std::endl does not only insert a newline character – it also flushes the stream buffer every time it is called. In practice this means that instead of accumulating output in memory and writing it in large efficient chunks, the program repeatedly forces the operating system to perform tiny writes. Those writes are expensive because they involve syscalls. Modern systems are optimized for throughput via buffering and std::endl completely defeats that optimization.

The performance difference can be enormous. Look at the following code which compares writing one million lines to file using std::endl versus using '\n':

C++
#include <chrono>
#include <iostream>
#include <fstream>

constexpr unsigned int kNumberOfWrites {1000000U};

int main() {
    std::chrono::milliseconds std_endl_duration;
    std::chrono::milliseconds backslash_n_duration;

    {
        std::ofstream file("endl.txt");
        const auto start = std::chrono::high_resolution_clock::now();

        for (unsigned int i=0U; i<kNumberOfWrites; i++) {
            file << i << std::endl;
        }

        const auto end = std::chrono::high_resolution_clock::now();
        std_endl_duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    }

    {
        std::ofstream file("newline.txt");

        const auto start = std::chrono::high_resolution_clock::now();

        for (unsigned int i=0U; i<kNumberOfWrites; i++) {
            file << i << '\n';
        }

        const auto end = std::chrono::high_resolution_clock::now();
        backslash_n_duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);

    }

    std::cout << "endl: " << std_endl_duration.count() << " ms" << std::endl;
    std::cout << "\\n: " << backslash_n_duration.count() << " ms" << std::endl;
}

On my machine the result is:

  • std::endl: 1387 ms
  • \n: 47 ms

This is nearly a 30× slowdown caused entirely by unnecessary flushing.

Why I chose to write to file in this example instead of just writing to terminal? If the benchmark used std::cout, the difference would be much smaller and sometimes almost invisible. The reason is that terminals are already extremely slow and often line-buffered by default, meaning that writing '\n' may trigger a flush anyway. In that case the benchmark would mostly measure the terminal rendering speed rather than flushing overhead. Files are fully buffered by the OS, which exposes the real cost of std::endl.

Does this mean you should stop using std::endl in favor of \n? The answer is: no. In many programs the performance difference simply does not matter. If your application prints a few status messages, logs occasional errors or saves some data to file from time to time, using std::endl is completely fine and can even be desirable because the explicit flush guarantees that the output appears immediately.

AI is powerful. Snippets are instant.

Stop prompting for the same patterns repeatedly. Get almost 100 free VS Code snippets for C++, Python, CMake and Bazel from piko::snippets GitHub repository.

Branchless logic

Another excellent example of AI-generated C++ code that looks highly optimized but performs terribly in practice is unnecessary branchless programming. Large language models seem to be sometimes biased towards generating branchless code, most likely because “branchless” often appears in discussions about performance optimization, SIMD and CPU pipeline efficiency. Branches, on the other hand, are often described as expensive due to branch prediction misses. As a result, from time to time an ordinary conditional logic is being generated as an arithmetic expressions such as:

C++
sum += condition * expensive();

instead of simply writing:

C++
if (condition) {
    sum += expensive();
}

The problem is that this optimization completely ignores what is actually worth to skip – it suggests skipping if and leaves expensive() call being executed on every single iteration, even when the condition is false. Take a look at the following code:

C++
#include <chrono>
#include <cmath>
#include <iostream>

constexpr unsigned int kNumberOfIterations {1000000U};

double expensive()
{
    double x {};
    
    for (unsigned int i=0U; i<100U; i++) {
        x += std::sqrt(i);
    }

    return x;
}

int main() {
    std::chrono::milliseconds branchless_duration;
    std::chrono::milliseconds branch_duration;

    {
        const auto start = std::chrono::high_resolution_clock::now();
        double sum {};

        for (unsigned int i=0U; i<kNumberOfIterations; i++) {
            const bool condition = ((i % 100) == 0);
            sum += static_cast<int>(condition) * expensive();
        }

        const auto end = std::chrono::high_resolution_clock::now();
        branchless_duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    }

    {
        const auto start = std::chrono::high_resolution_clock::now();
        double sum {};

        for (unsigned int i=0U; i<kNumberOfIterations; i++) {
            if ((i % 100) == 0) {
                sum += expensive();
            }
        }

        const auto end = std::chrono::high_resolution_clock::now();
        branch_duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    }

    std::cout << "Branchless: " << branchless_duration.count() << "ms" << std::endl;
    std::cout << "Branch: " << branch_duration.count() << "ms" << std::endl;
}

The result on my machine is:

  • Branchless: 406ms
  • Branch: 5ms

Here, it is clearily visible that the branchless version simply does 100 times more work (i.e. 100 more expensive() calls) than the branched version.

Does this mean branchless programming should never be used? Of course: no. Branchless techniques can be extremely effective when the condition itself is unpredictable (e.g. based on random data) and both execution paths are cheap. In those situations, branch mispredictions can dominate the execution time.

Preferring function objects over templates

The next common pattern in AI-generated C++ code is overusing std::function in places where templates would be significantly faster. I think this happens because large language models strongly associate std::function with “modern” and “clean” C++ design.

Consider the following code:

C++
#include <chrono>
#include <functional>
#include <iostream>

constexpr unsigned int kNumberOfIterations {1000000U};

int square(const int x)
{
    return x * x;
}

int run_std_function(const std::function<int(int)>& callback)
{
    int sum {};

    for (unsigned int i=0U; i<kNumberOfIterations; i++) {
        sum += callback(i);
    }

    return sum;
}

template<typename CallbackType>
int run_template(CallbackType callback)
{
    int sum {};

    for (unsigned int i=0U; i<kNumberOfIterations; i++) {
        sum += callback(i);
    }

    return sum;
}

int main()
{
    std::chrono::milliseconds std_function_duration;
    std::chrono::milliseconds template_duration;

    {
        const auto start = std::chrono::high_resolution_clock::now();
        std::ignore = run_std_function(std::function<int(int)>(square));
        const auto end = std::chrono::high_resolution_clock::now();

        std_function_duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    }

    {
        const auto start = std::chrono::high_resolution_clock::now();
        std::ignore = run_template(square);
        const auto end = std::chrono::high_resolution_clock::now();

        template_duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    }

    std::cout << "std::function: " << std_function_duration.count() << "ms" << std::endl;
    std::cout << "template: " << template_duration.count() << "ms" << std::endl;
}

The result on my machine is:

  • std::function: 32ms
  • template: 1ms

This is a dramatic difference especially that unlike in the earlier branchless-vs-branch example, where one version actually performed more computation, here both versions execute exactly the same logical work.

The reason comes from how std::function is implemented internally. std::function uses type erasure, meaning that the concrete callable type is hidden behind a generic runtime mechanism. Take a look at the following code:

C++
struct Functor {
    int operator()(int x) const { return x + 1; }
};

int main()
{
    auto lambda = [](int x) { return x + 1; };
    Functor functor;

    auto lambda_obj = std::function<int(int)>(lambda);
    auto functor_obj = std::function<int(int)>(functor);

    std::cout << lambda_obj(11) << std::endl;
    std::cout << functor_obj(11) << std::endl;
}

lambda and functor are completely different compile-time types and yet std::function can store both of them, but unfortuantely the mechanism allowing to do that (i.e. type erasure) comes with a runtime cost.

The template version is completely different. Since the callback type is known at compile time, the compiler can generate proper implementation and thus it comes with no runtime cost.

This example is particularly interesting because the slower version genuinely looks more advanced. In many discussions on the Internet a lot of people (and thus AI systems now) for years tended to prefer abstractions that appear architecturally elegant:

  • generic wrappers
  • runtime polymorphism
  • decoupled interfaces
  • reusable callback systems

And all this is often fine, but not when performance is critical.

Does this mean that you should never use std::function? Again: no. In many applications its overhead is completely negligible compared to the flexibility it provides. std::function becomes particularly useful when callables need to be stored, passed around dynamically or treated uniformly at runtime – something templates fundamentally cannot do on their own because every template instantiation has a different concrete type.

A very common example of such situation is storing multiple callbacks inside a container. The std::vector of std::function is often the way to go because you can throw in a lambda, a functor, a function pointer etc. and as long as their signatures match the std::function template type, everything will work.

Read also: