Posted in

When AI gets C++ bitmask enums almost right

Today I want to take a look at a piece of C++ code that Copilot generated for me recently. The code here will not be 1:1 the same because originally it was inserted as a part of the bigger code base – I changed the names and simplified it to focus on the key topic, but the overall idea is exactly the same.

At first glance, the generated code looked perfectly reasonable. In fact, it even compiled. The problem is that it also contained a subtle semantic trap that is very easy to miss if you don’t stop and think carefully about what the code is actually meant to do.

The code started with an enum representing permissions where the values are bitmasks:

C++
enum class Permission
{
    kNone = 0,
    kRead = 1 << 0,
    kWrite = 1 << 1,
    kExecute = 1 << 2
};

And then Copilot generated an associated stream operator to print the enum value using std::cout:

C++
std::ostream& operator<<(std::ostream& os, const Permission p)
{
    switch (p) {
        case Permission::kNone:
            return os << "None";
        case Permission::kRead:
            return os << "Read";
        case Permission::kWrite:
            return os << "Write";
        case Permission::kExecute:
            return os << "Execute";
        default:
            return os << "UNKNOWN PERMISSION";
    }
}

So far, everything still looks fine. The next step overloads the bit or operator for the Permission enum:

C++
Permission operator|(const Permission lhs, const Permission rhs)
{
    return static_cast<Permission>(static_cast<int>(lhs) | static_cast<int>(rhs));
}

Nothing immediately suspicious here either, but the things start to look interesting because Copilot generated also a function that, according to some complex logic, returns combinations of permissions in form of:

C++
Permission GetPermissions()
{
    return Permission::A | Permission::B;
}

This is where things start getting tricky. The code is not obviously wrong because representing flags as bitmasks is a completely valid technique, but let’s test this with a simple example:

C++
Permission GetPermissions()
{
    return Permission::kNone | Permission::kWrite;
}

int main()
{
    const Permission p = GetPermissions();
    std::cout << "Current permissions: " << p;
}

This works perfectly fine because:

MDX
00000000 | 00000010 = 00000010

And 00000010 corresponds exactly to Permission::kWrite, so the output becomes:

MDX
Current permissions: Write

However, when the function reaches a path like this:

C++
Permission GetPermissions()
{
    return Permission::kRead | Permission::kWrite;
}

the output suddenly changes to:

MDX
Current permissions: UNKNOWN PERMISSION

How is this possible that although all enum values of the Permission enum are covered in switch/case statement, after getting the object p of type Permission and printing it, no valid value of the enum is displayed?

The answer is: the value returned from GetPermissions() is simply not represented by any enum field. Take a look:

MDX
Permission::kRead  = 00000001
Permission::kWrite = 00000010

Their bitwise or produces:

MDX
00000011

But there is no enum field equal to 00000011 and C++ does not forbid such value from existing inside an enum object.

The key problem

This reveals a deeper conceptual conflict:

  • enums are designed to represent one value from a closed set
  • bitmasks are designed to represent combinations of values

These are fundamentally different concepts and without writing a logic which actually enforces the rules of how these 2 contradicting worlds co-exist, we end up with a subtle bug hidden behind a clean API. Of course, the solution is not to keep extending the enum with values like:

MDX
kReadWrite
kWriteExecute
kReadExecute

because this does not scale. With more flags, the number of combinations grows exponentially. Let’s look at how to fix the issue properly.

AI is powerful. Snippets are instant.

Stop prompting for the same patterns repeatedly. Get almost 100 free VS Code snippets for C++, Python, CMake and Bazel from piko::snippets GitHub repository.

Don’t lie with the function signature

Using enums to represent individual bitmask values is perfectly fine. It improves the usage and the readability because you no longer need to remember whether “read” permission is equal to 1, 2, or 16. But the moment you start combining those values, the result is no longer guaranteed to be a valid Permission enum value, so the API should reflect that reality.

The first step is changing the overloaded | operator – instead of pretending that the result is still a Permission, return an integer mask explicitly:

C++
int operator|(const Permission lhs, const Permission rhs)
{
    return static_cast<int>(lhs) | static_cast<int>(rhs);
}

Now the operator no longer falsely suggests that combining two permissions produces another valid enum field. Similarly, any function returning combined permissions should return an integer mask instead of Permission:

C++
int GetPermissions()
{
    return Permission::kRead | Permission::kWrite;
}

int main()
{
    const int p = GetPermissions();
    std::cout << "Current permissions: " << p;
}

This now produces:

MDX
Current permissions: 3

The value is no longer pretending to be a single enum value. It is simply a bitmask containing multiple flags what is clearily communicated by the API now. If you want, you can print it in a nicer form by using a helper function – here is an example of a simple one:

C++
void PrintPermissions(const int p)
{
    if (p & static_cast<int>(Permission::kRead)) {
        std::cout << "Read";
    }

    if (p & static_cast<int>(Permission::kWrite)) {
        std::cout << "Write";
    }

    if (p & static_cast<int>(Permission::kExecute)) {
        std::cout << "Execute";
    }

    std::cout << std::endl;
}

Now the output becomes:

MDX
Current permissions: ReadWrite

Read also: