Welcome to the next pikoTutorial !

Recently I’m experimenting with different ways of error handling in Python and I stumbled upon a use case in which it would be very helpful to create an enum by deriving from a different enum.

Use case

Let’s say we design an interface for a class with multiple member functions, each returning some error code:

class Client:
    def execute_transaction() -> ErrorCode:
        pass
    def withdraw_money() -> ErrorCode:
        pass
    ...

The common approach is to gather all the error codes in a single enum type like the following:

class ErrorCode(Enum):
    INVALID_TOKEN = 0
    CONNECTION_DROPPED = 1
    INVALID_PRICE = 2
    OUT_OF_FUNDS = 3

However, if your interface exposes many methods, such ErrorCode type quickly becomes a mess, embracing all the possible error codes that different domains of the system may report.

The use case I wanted to try out is to split the ErrorCode type into types dedicated for the individual domains and their specific errors, so that the interface looks more like this:

class Client:
    def execute_transaction() -> TransactionError:
        pass
    def withdraw_money() -> AccountError:
        pass
    ...

TransactionError and AccountError must be defined in separate enums:

class TransactionError(Enum):
    INVALID_TOKEN = 0
    CONNECTION_DROPPED = 1
    INVALID_PRICE = 2

class AccountError(Enum):
    INVALID_TOKEN = 0
    CONNECTION_DROPPED = 1
    OUT_OF_FUNDS = 2

The problem now is that the generic errors like INVALID_TOKEN and CONNECTION_DROPPED can happen when calling both methods, so they must be repeated in both of the enums. What if I need to change the name of the INVALID_TOKEN field? I had to do that in both of them. What if I need to change the value of CONNECTION_DROPPED field? I also had to do that in both of them. What if I forgot about one of the enums? I have a problem – problem which scales proportionally to the number of error domains and remember that here we’re assuming the use case in which there is plenty of different domains.

One could say that the above problem is related only to the maintainability of the code, but unfortunately there’s a deeper issue which impacts the usability of such approach. If execute_transaction() returns TransactionError.CONNECTION_DROPPED error and withdraw_money() returns AccountError.CONNECTION_DROPPED error, both of these values are intended to represent exactly the same error. However, if you compare them, Python will say that they are not equal:

conn_dropped_1 = TransactionError.CONNECTION_DROPPED
conn_dropped_2 = AccountError.CONNECTION_DROPPED
print(conn_dropped_1 == conn_dropped_2)  # prints False

And the worst part is that this is expected behavior – quick look at the underlying types of both objects reveal that they actually are not equal:

print(type(conn_dropped_1))  # prints "<enum 'TransactionError'>"
print(type(conn_dropped_2))  # prints "<enum 'AccountError'>"

In order for such objects to be equal, they would have to share some base type which would own the common error codes. The domain-specific types would derive from it adding they domain-specific error codes. This would resolve both maintainability problem and the comparison problem.

class GenericError(Enum):
    INVALID_TOKEN = 0
    CONNECTION_DROPPED = 1

class TransactionError(GenericError):    
    INVALID_PRICE = 2

class AccountError(GenericError):
    OUT_OF_FUNDS = 2

Here we however hit the wall because Python does not allow for the derivation from enumeration types. I needed some kind of an “expandable enum” which would allow for derivation and which would keep the type consistency across such hierarchy of enums. And because it sounded like a good exercise, I decided to implement such expandable enums for my use case.

Expandable enum definition

Let’s first define what would be the properties of such expandable enum type, especially their:

• types across the hierarchy
• rules of enum field definitions
• fields equality
• per-enum field types

Types across the hierarchy

First of all, I want to be able to use such type as every other enumeration type, what means that I want every field of my expandable enum type to be of type of the expandable enum type itself. Quick inspection of the default Python enum shows the mentioned properties:

• enum.EnumType is a metaclass of type T
• field T.F is of type T

class T(Enum):  
    F = 0  
  
print(type(T))        # prints "<class 'enum.EnumType'>"
print(type(T.F))      # prints "<enum 'T'>"
print(type(T.F) == T) # prints "True"

Assuming that expandable enum type E has field F, the generalized requirement written in Python (will be useful for unit tests) looks like below:

assert type(E) == ExpandableEnumType
assert type(E.F) == E

print(type(E))   # prints "<class 'ExpandableEnumType'>"
print(type(E.F)) # prints "<enum 'E'>"

The exact type of the field will depend on the actual type where the field has been defined. It means that if there’s an expandable enum type B which has field F1 and another expandable enum type E which derives from B and defines field F2, the type of both B.F1 and E.F1 will be B, but the type of E.F2 will be E:

class B(metaclass=ExpandableEnumType):
    F1 = 0

class E(B):
    F2 = 1

assert type(B) == ExpandableEnumType
assert type(E) == ExpandableEnumType
assert type(B.F1) == B
assert type(E.F1) == B
assert type(E.F2) == E

print(type(B.F1)) # prints "<enum 'B'>"
print(type(E.F1)) # prints "<enum 'B'>"
print(type(E.F2)) # prints "<enum 'E'>"

This is probably the most important assumption because this trait will resolve the problem that we encountered when comparing TransactionError.CONNECTION_DROPPED to AccountError.CONNECTION_DROPPED. The fact of accessing F1 via derived type E should not change its underlying type. In other words, I want field CONNECTION_DROPPED to remain GenericError regardless of whether I access it through GenericError, TransactionError or AccountError type.

Rules of enum field definitions

As soon as we start deriving one enum from another, we stumble upon the problem on how to treat the situation when an enum subclass defines a field which is equal to one of the fields defined in its base class. Additionally, what does it mean for an enum field to be equal to another one? Is FIELD = 0 equal to FIELD = 1? Is FIELD = 0 equal to OTHER = 0?

First things first – enumeration field has a name and a value. In case of AccountError defined above and its only field, OUT_OF_FUNDS is the name and 2 is its value. To get rid of any confusion which could arise while using expandable enums, there must be a rule that none of the fields defined across the enum hierarchy may share common name or value. If it happens, the field will be considered as conflicting. According to this rule, the following type is invalid due to duplicated field name F1:

class B(metaclass=ExpandableEnumType):
    F1 = 0x23

class E(B):
    F1 = 0x24

And the following type is invalid due to duplicated value 0x23:

class B(metaclass=ExpandableEnumType):
    F1 = 0x23

class E(B):
    F2 = 0x23

Without introducing this rule, it would be possible to achieve very disturbing constructs. For example, it would be allowed to define the following hierarchy:

class GenericError(metaclass=ExpandableEnumType):
    CONNECTION_DROPPED = 0

class TransactionError(GenericError):    
    CONNECTION_DROPPED = 1

What would be the type of TransactionError.CONNECTION_DROPPED? GenericError or TransactionError? This is the reason why it is enough for one of the two (name or value) to be duplicated to consider the whole enum hierarchy as invalid.

Fields equality

At this point you may start asking yourself a question: why is there an alternative (or) in the above condition? Shouldn’t the field be considered as a conflicting if it’s equal to some other field? Shouldn’t equality of two fields include name and value?

Remember that in the previous paragraph we were talking about the rules of defining fields across the enum hierarchy, not about the rules of equality of two fields. Two fields are considered to be equal if their names and values and types are the same. When dealing with inheritance, the following must be fulfilled:

class T:
    F1 = 0

class E(T):
    F2 = 1

assert E.F1 == T.F1

Per-enum field types

There’s one more capability that expandable enums can bring – after allowing an enum type to have a hierarchy, it is technically possible for each of the hierarchy levels to have its own field type. Look at this theoretical example:

class Rectangle:
    def get_area(self) -> int:
        return self.value[0] * self.value[1]

class Point:
    def get_coordinates(self) -> str:
        return f"x = {self.value[0]}, y = {self.value[1]}"

class B(Rectangle, metaclass=ExpandableEnumType):
    F1 = (10, 12)

class E(B, Point):
    F2 = (5, 12)

assert B.F1.get_area() == 120
assert E.F1.get_area() == 120
assert E.F2.get_coordinates() == "x = 5, y = 12"

It is an interesting property, however I don’t think it will be a good practice to do such things when using expandable enums.

Implementation

Let’s start with a class diagram embracing all the rules that I’ve describe above. For simplicity, to not repeat every time “expandable enum type”, I called this project ExpanEn (Expandable Enum).

Quick description of the relations:

• class ExpandableEnumType is the key element – it will be used as a metaclass of every user defined enum and will determine the behavior that I expect from the expandable enums implementing custom logic in __new__ and __setattr__ functions
• class Expanen is almost empty because its only responsibility is to provide a convenient interface for the user – instead of writing metaclass=ExpandableEnumMeta in every user defined type, user will be able to just derive from Expanen and extend it with enum fields
• class ExpanenField represents a single enum field (name + value). It will be a parent of every user defined enum
• UserDefinedEnumType may optionally derive from some custom field type to extend the functionalities of an enum field
• ConflictingEnum is just an exception to raise when conflicting fields are detected in the enum hierarchy

ExpanenField

As the first one, let’s create ExpanenField because it will be necessary during ExpandableEnumType class implementation:

class ExpanenField:  
    def __init__(self, name: str, value: any):  
        self.name: str = name  
        self.value: any = value  
  
    def __str__(self) -> str:  
        return f"({self.name}: {self.value})"

ExpandableEnumType

Implementation of ExpandableEnumType is focused mainly on the __new__ function:

class ExpandableEnumType(type):  
    def __new__(cls, name, bases, dct):  
        new_class = super().__new__(cls, name, bases, dct)  
        # gather fields which already exist in the base classes
        # of the type which is currently being created
        existing_fields = {field for base in bases
                                 for field in base.__dict__.values()
                                 if isinstance(field, ExpanenField)}  
        # iterate over class members in the type which is currently
        # being created and check if the conflicting fields exists
        # higher in the hierarchy - if yes, raise the exception
        for name, value in dct.items():
            # omit special member functions
            if name.startswith("__"):  
                continue
            for field in existing_fields:
                # extract raw field's name from the full name being
                # "Type.Field"
                raw_field_name: str = field.name.split(".")[-1]  
  
                if raw_field_name == name or field.value == value:  
                    raise ConflictingEnumField(f"Failed to create an expandable enum {new_class.__name__} due to "                                               f"duplicated enum field! {new_class.__name__}.{name}: {value} conflicts "                                               f"with {field.name}: {field.value}")  
            # if the field is unique, convert the value assigned to
            # the class member to ExpanenField type by calling its
            # constructor
            cls.__setattr__(new_class, name, value)  
            existing_fields.add(new_class(name, value))  
  
        return new_class  
  
    def __setattr__(cls, name, value):
        # expand the raw field's name, so that it contains also the
        # class name (it will display as "Type.Field" instead of
        # just "Field")
        super().__setattr__(name, cls(f"{cls.__name__}.{name}", value))  
  
    def __repr__(cls):  
        return f"<enum '{cls.__name__}'>"

Expanen

As mentioned earlier, Expanen class is responsible for delivering a convenient interface, so its implementation is very simple:

class Expanen(ExpanenField, metaclass=ExpandableEnumType):  
    pass

Example usage

Since the implementation is ready, let’s now look at the example usage of such expandable enum construct. Let’s try to recreate the situation from the beginning of this article. First I define a custom field type of my error enum:

class ErrorModel:  
    def code(self) -> int:  
        return self.value.split(": ")[0]  
  
    def description(self) -> str:  
        return self.value.split(": ")[1]

Now, let’s re-define the errors so that they can derive from each other:

class GenericError(Expanen, ErrorModel):  
    SUCCESS = "0: Operation successful"  
    INVALID_TOKEN = "387: Token has expired"  
    CONNECTION_DROPPED = "172: Connection has been established, but \
                          then dropped due to API policy violation"  
  
class TransactionError(GenericError, ErrorModel):  
    INVALID_PRICE = "948: Failed to execute transaction due to invalid price"  
  
class AccountError(GenericError, ErrorModel):  
    OUT_OF_FUNDS = "257: Withdrawal requested, but there is no enough money"

The interface of the client class could look like this:

class Client:  
    def execute_transaction(self) -> TransactionError:  
        pass  
  
    def withdraw_money(self) -> AccountError:  
        pass

And finally the example usage:

try:  
    error = Client().execute_transaction()  
  
    if error == error.CONNECTION_DROPPED:  
        print(f"{error.description()} - reconnecting...")  
    if error in (TransactionError.INVALID_TOKEN,
                 TransactionError.INVALID_PRICE):  
        raise RuntimeError(error)  
except Exception as e:  
    print(f"Unable to recover - {e}")

For me, it looks like such expandable enumeration type could really benefit when combined with Result object, but that’s a topic for another article.

GitHub repository

The project is available on GitHub, so feel free to clone and test it out.