2.9 Magic methods and operator overloading

What this is and why it exists

Magic methods are how your own objects plug into Python's own syntax: how len(x) knows what to return, how x == y decides, how for item in x works. Implementing a few of them well makes debugging and testing dramatically nicer, and it is also how the libraries ahead do their apparent magic — a PyTorch dataset is an object that answers indexing and length, and nothing more mysterious than that.

The vocabulary

• Magic (or dunder) method — a method with two underscores on each side, called by the language rather than by you.
• Representation — what repr(x) returns: the developer-facing form, ideally something you could paste back in.
• Display string — what str(x) returns: the user-facing form.
• Equality — what == uses.
• Hash — the number used to place an object in a set or as a dict key.
• Sequence protocol — length and indexing, which together make an object behave like a list.
• Iterator protocol — producing items one at a time for a for loop.
• Callable — an object that can be used like a function.

The mental model

Python's syntax is a set of agreements. When you write len(x), Python calls the object's length method; when you write x[0], it calls the indexing method; when you write with x: it calls the enter and exit methods. Implementing the agreement makes your object work everywhere that syntax is accepted — which is the point, and also the limit: implement it only where it genuinely helps a reader, because an object that redefines + to mean something surprising is worse than one with a clearly-named method.

The two worth writing almost always are representation and equality. A good representation is what you see when you print a list of your objects at three in the morning, and the convention is that it should look like the call that would recreate the object. Without one you get a class name and a memory address, which tells you nothing. Equality lets == and in and assert result == expected behave as tests expect — and defining it is where this topic's trap lives.

Define equality and you must think about hashing. Python's rule is that objects which compare equal should hash the same, and defining equality without hashing makes your class unhashable — so it stops working in sets and as a dict key, quietly, wherever somebody tried. Two clean ways out: define both, hashing exactly the fields equality uses; or use a frozen dataclass from the next topic, which does both for you correctly and is the answer most of the time.

The sequence agreement — length plus indexing — is worth knowing because it is what the libraries expect. Implement those two and your object can be measured, indexed, sliced by anything that asks, and used by a data loader. The iterator agreement is separate: an iterable returns a fresh iterator each time, and an iterator produces items until it raises the stop signal. The glossary is precise about why that distinction matters: a container "produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop", while an iterator hands back the same exhausted object — "making it appear like an empty container". That sentence explains a bug you will otherwise meet by surprise in the generators topic.

Context managers and callables round it out. Enter and exit make your object usable in a with block, which is how you give a resource of your own the same guarantee a file gets. And a call method makes an instance usable like a function while still holding state — which is exactly what a configured transformer or a loss function is.

In code

Checked against the data-model and glossary references.

class ScoreSet:
    def __init__(self, subject, scores):
        self.subject = subject
        self.scores = list(scores)

    def __repr__(self):
        return f"ScoreSet({self.subject!r}, {self.scores!r})"

    def __eq__(self, other):
        if not isinstance(other, ScoreSet):
            return NotImplemented
        return (self.subject, self.scores) == (other.subject, other.scores)

    def __hash__(self):
        # Equal objects must hash equally; hash exactly what equality used.
        return hash((self.subject, tuple(self.scores)))

    def __len__(self):
        return len(self.scores)

    def __getitem__(self, index):
        return self.scores[index]

    def __iter__(self):
        return iter(self.scores)


s = ScoreSet("maths", [88, 71, 94])
print(len(s), s[0], max(s))
print(s == ScoreSet("maths", [88, 71, 94]))

Two details are deliberate. Equality returns the not-implemented sentinel rather than False when the other object is a different type, which lets Python try the comparison the other way round instead of asserting an answer it has no business giving. And the hash uses a tuple of the scores, because a list cannot be hashed — which is the same immutability rule from the containers topic showing up again.

What you should now be able to explain or do

Write a representation that would recreate the object, and say why it beats the default. Define equality and hashing together, and state the rule connecting them. Say what happens to a class that defines equality alone. Implement length and indexing and say which libraries then accept your object. Explain, from the glossary's own words, why iterating an iterator twice looks like an empty container.

Check yourself