P-1.3 Numbers, Text, and Why They Go Wrong

What this is and why it exists

Two families of defect follow programmers through an entire working life, and both come from one place: a finite machine representing infinite things.

Whole numbers have a range, and arithmetic that leaves it wraps around. Decimal numbers are stored as near neighbours of what you wrote, so calculations that should agree do not quite. Text is the worst of the three. Bytes and characters look identical, right up until somebody types a name your program was never tested on.

Learning this takes an afternoon. Learning it from a production incident takes rather longer, and somebody else usually pays for the lesson.

The vocabulary

• Integer — a whole number stored in a fixed number of bits.
• Overflow — what happens when arithmetic leaves the representable range.
• Floating point — the usual way of storing numbers with fractional parts.
• Rounding error — the gap between the number you wrote and the number stored.
• Tolerance — a deliberate margin within which two results count as equal.
• Byte — eight bits of storage. Not a character.
• Character — one written symbol, which may take several bytes.
• Encoding — the agreement about which bytes stand for which characters.

The mental model

Start with whole numbers. A fixed number of bits holds a fixed range, and arithmetic that leaves that range does not raise its hand in every language. It quietly produces a wrong answer, and often a negative one, because the highest bit that would have carried is the one that means "negative". A counter that has run for long enough becomes a bug with no error message attached.

Decimals are different and, once seen, less mysterious. Most decimal fractions have no exact representation in binary. A tenth does not, in the same way a third has no exact decimal. So what gets stored is a very near neighbour of what you wrote. The important word is systematic. The error is not random noise; it is predictable, which is precisely what makes it possible to reason about rather than fear.

That gives the rule about comparison. Two calculations that should agree mathematically can differ in their last bits, so an equality check fails on two numbers that are both correct. Asking whether two decimal results are equal is the wrong question. The right one is whether they differ by less than a tolerance you chose on purpose, for reasons you can state.

Then text, which is where most of the real damage happens. A byte is eight bits. A character is a written symbol. They are not the same thing and never were. For a long time a lot of software behaved as if they were. That software was only ever tested on plain unaccented English.

An encoding is the agreement about which bytes stand for which characters. Text handled without knowing its encoding is bytes pretending to be words. This is exactly where names break, where currency symbols break, and where text in Indian languages breaks. Not in exotic cases, but in the ordinary case of a real person's real name.

The fastest way to own all of this is to produce it. Write the smallest program that wraps an integer and the smallest one that fails a decimal comparison. Having made both happen yourself, you will recognise their shape inside something much larger, years later, when nothing is labelled.

What you should now be able to explain or do

Explain why a whole number wraps and why the result is often negative. Explain why a stored decimal is a near neighbour of the number you wrote, and why that error is systematic rather than random. Say why comparing two decimal results for equality is the wrong test, and what to do instead. State the difference between a byte and a character, and say what an encoding is for. Name where text handling actually breaks in practice. Reproduce one overflow and one rounding surprise, and account for both.

Check yourself