Software Mistakes and Tradeoffs

Contents

• 01Introduction
• 02Code Duplication Is Not Always Bad
• 03Exceptions vs. Other Error-Handling Patterns
• 04Balancing Flexibility and Complexity
• 05Premature Optimization vs. Optimizing the Hot Path
• 06Simplicity vs. Cost of Maintenance for Your API
• 07Working Effectively With Date and Time Data
• 08Leveraging Data Locality and Memory of Your Machines
• 09Third-Party Libraries: Libraries You Use Become Your Code
• 10Consistency and Atomicity in Distributed Systems
• 11Delivery Semantics in Distributed Systems
• 12Managing Versioning and Compatibility

Introduction

Every meaningful software decision is a tradeoff that forecloses other paths. The chapter previews recurring tensions — testing strategy, code patterns, architecture — and argues engineers must weigh competing forces consciously rather than apply rules mechanically.

Decisions Limit Future Evolution

The longer a system lives, the harder it becomes to back out of earlier design decisions. Tradeoff awareness is what keeps optionality alive — the wrong choice locked in early compounds into expensive migrations later.

Unit vs. Integration Test Proportions

Heavy unit testing is fast and isolated but misses integration bugs; integration tests catch real wiring but are slower and more brittle. Neither extreme is correct — the right mix depends on deployment risk and feedback-speed needs.

Patterns Are Context-Dependent

GoF code patterns and architecture patterns all carry costs. Applying Singleton, Strategy, microservices, or event-driven architecture outside their sweet spot adds complexity without payoff.

Microservices Buy Scale With Complexity

Microservices enable independent scaling and deployment but introduce network, consistency, and operational overhead. They are appropriate only when the organization and load justify the cost.

Code Quality Is Multidimensional

Readability, testability, modifiability, and performance often pull against each other. "Quality" must be defined per project rather than treated as a universal absolute.

Time-to-Market and SLAs Constrain Design

External pressures — deadlines, contractual availability targets, SLAs — often dictate which tradeoff is acceptable. The "best" technical answer loses to the shippable answer when the calendar bites.

Tradeoff

A decision that gains in one dimension at the cost of another.

Unit test

A test exercising a single component in isolation, typically without I/O.

Integration test

A test exercising multiple components together, including external systems.

Test pyramid

Heuristic test mix — many unit tests, fewer integration, very few end-to-end.

Design pattern

A reusable solution template for a recurring code-level problem.

Architecture pattern

A system-level structural choice (monolith, microservices, event-driven).

Scalability

Capacity to handle growing load by adding resources.

SLA

Contractual guarantee for metrics like availability or latency.

Multiple choice

A startup with five engineers is deciding between a monolith and microservices for a brand-new product with uncertain requirements and modest expected load. Which framing is most consistent with this chapter?

• AMonolith, because microservices add network, consistency, and operational overhead that are unjustified at this team and load size.
• BMicroservices, because they are the modern default and scale better.
• CMicroservices, because they will be impossible to introduce later.
• DMonolith, because microservices always cost more than they deliver.

Spot the issue

A team mandates a fixed ratio: 90% unit tests, 10% integration tests, no exceptions, regardless of project. What's the mistake?

• AThe pyramid is wrong; integration tests should always dominate.
• B90/10 is fine, but the team should also write end-to-end tests.
• CUnit tests shouldn't be counted at all.
• DTreating a heuristic test mix as a universal rule ignores that the right mix depends on deployment risk and feedback-speed needs.

Multiple choice

Which of the following is the strongest sign that "code quality" is being treated naively on a project?

• AThe team measures cyclomatic complexity per module.
• BReadability, performance, and modifiability are debated as tradeoffs each sprint.
• CThe team treats "quality" as a single universal absolute, applied identically to a prototype and a payment processor.
• DDifferent services adopt different lint rules.

True / False

Once an early design decision is locked in, the cost of reversing it tends to compound as the system grows.

• TrueTrue
• FalseFalse

Code Duplication Is Not Always Bad

The chapter pushes back on dogmatic DRY, arguing premature abstraction couples unrelated code paths and slows delivery, especially across service boundaries. The right question is not "is this duplicated?" but "do these call sites have the same reason to change?"

DRY Can Hide Complexity, Not Remove It

Forcing two superficially similar code paths into a single abstraction couples them. When their requirements later diverge, the abstraction breaks down and the cost of untangling exceeds the original duplication.

Duplication Across Codebases vs. Within One

Removing duplication inside one service is usually cheap. Removing it across independently deployed services drags in coordination, versioning, and release-cycle costs that often exceed the savings.

Shared Libraries Cause Version Coordination Pain

When N services depend on a shared library, a breaking change forces synchronized upgrades. Even non-breaking changes create lock-step deployment pressure that erodes service autonomy.

Microservice Extraction Is a Heavy Hammer

Extracting common logic into its own service adds network calls, failure modes, and availability dependencies. The operational complexity is often worse than tolerating the original duplication.

Loose Coupling Can Be Bought With Duplication

Two services owning their own copy of similar logic remain free to evolve at different speeds. Autonomy is frequently more valuable than a single canonical implementation.

Same Code, Different Reasons to Change

Apparent duplication may serve distinct business domains. Merging them violates the single-responsibility intent and produces brittle abstractions that fight every future change.

Premature Abstraction Beats Premature Duplication

It is easier to deduplicate later, once the real shape is known, than to untangle a wrong abstraction. Wait for the third occurrence before extracting.

DRY

Principle stating every piece of knowledge should have a single authoritative representation.

Tight coupling

Components depend strongly on each other's internals and must change together.

Loose coupling

Components interact through narrow, stable interfaces and evolve independently.

Shared library

A packaged module of code depended on by multiple services or applications.

Microservice extraction

Refactoring shared logic into a separately deployed service.

Abstraction

A generalized interface that hides implementation differences behind a common surface.

Delivery speed

How quickly a team can ship changes, often degraded by coordination costs.

Spot the issue

Two independently deployed services, Billing and Reporting, each have a 40-line function that formats currency values for their own domain. A platform engineer extracts the function into a shared library used by both services. Six months later, Billing needs to round half-up for tax compliance while Reporting still needs banker's rounding. What's the underlying mistake?

• AThe shared library should have been a microservice instead.
• BBanker's rounding is wrong for reporting.
• CThe two functions had different reasons to change; deduplicating across service boundaries coupled unrelated business domains and now blocks divergent evolution.
• DBilling should be merged into Reporting.

Multiple choice

What is the strongest argument for tolerating duplication across two independently deployed microservices?

• AEach service retains autonomy and can evolve its copy at its own pace without coordinating releases.
• BWriting code twice is faster than writing it once.
• CDuplication catches more bugs than abstraction does.
• DShared libraries are technically impossible across services.

Multiple choice

Which heuristic does the chapter offer for when extraction is actually justified?

• AExtract on the first occurrence to stay ahead of the curve.
• BExtract on the second occurrence — two examples is enough to see the pattern.
• CWait for the third occurrence so the real shape of the abstraction is visible.
• DNever extract; duplication is always cheaper.

Spot the issue

A platform team publishes a `common-utils` JAR depended on by 14 internal services. A "small" non-breaking change ships, and the team posts in #engineering: "Please redeploy your service this week so everyone is on v2.3." What does this reveal?

• AThe library should be versioned with SemVer.
• BThe library is too small to be worth maintaining.
• CThe 14 services should all be merged into one monolith.
• DShared libraries create lock-step deployment pressure even on non-breaking changes, eroding the autonomy of the consuming services.

Exceptions vs. Other Error-Handling Patterns

Error handling is fundamentally an API-design problem. The chapter surveys checked/unchecked exceptions, anti-patterns, third-party boundaries, concurrency, and the functional Try type — arguing the right mechanism depends on whether the error is recoverable and where it crosses boundaries.

Exception Hierarchy Design

A well-shaped hierarchy — specific exceptions inheriting from broader categories — lets callers catch at the right level of granularity instead of falling back to catch-all blocks that swallow unrelated failures.

Checked vs. Unchecked in Public APIs

Checked exceptions force callers to acknowledge failure modes (correctness aid, signature pollution); unchecked keep signatures clean but rely on documentation. Pick based on whether the error is recoverable and part of the contract.

Anti-Pattern — Exceptions as Control Flow

Throwing and catching to drive normal logic is slow and obscures intent. Expected outcomes belong in return types, not the exception channel.

Anti-Pattern — Swallowing Exceptions

Empty catch blocks and missed `try-with-resources` usage cause silent bugs and leaks. Original causes must be preserved via exception chaining when rethrowing.

Wrap Third-Party Exceptions at the Boundary

Translating external exceptions into your domain types prevents library types from bleeding into callers. It also protects you when you swap the dependency for a different implementation.

Exceptions Cross Thread Boundaries Poorly

In `ExecutorService` or `CompletableFuture` workflows, exceptions are captured into the future or lost unless you explicitly use `get`, `exceptionally`, `handle`, or an uncaught-exception handler.

The Functional Try<T> Type

`Try<T>` encodes success/failure in the value, making errors first-class data that flows through `map`/`flatMap`. It composes cleanly in streams and async chains where exceptions are awkward.

Checked exception

Exception the compiler forces callers to catch or declare.

Unchecked exception

RuntimeException subclass that propagates without compile-time enforcement.

Exception hierarchy

Inheritance tree of exception types organizing error categories.

Exception swallowing

Catching an exception without logging, rethrowing, or otherwise handling it.

Exception chaining

Preserving the original exception as the cause of a wrapping exception.

Try-with-resources

Java construct that auto-closes AutoCloseable resources even on exception.

Try<T> monad

Functional container holding either a success value or captured failure.

CompletableFuture

Java's async primitive; exceptions surface via exceptionally/handle/get.

Spot the issue

What's the mistake?

public Integer parseAge(String s) {
    try {
        return Integer.parseInt(s);
    } catch (NumberFormatException e) {
        return null;
    }
}
// Caller:
for (String s : userInputs) {
    Integer age = parseAge(s);
    if (age != null) { process(age); }
}

• AThe expected outcome — "input wasn't a number" — is being driven through the exception channel, which is slow and obscures intent; a `Try<Integer>` or `Optional<Integer>` return type expresses the contract more cleanly.
• B`NumberFormatException` should be checked.
• CThe caller forgot to log the exception.
• DThe method should be `static`.

Spot the issue

What's the mistake?

Future<Result> f = executor.submit(() -> doWork());
// later...
if (f.isDone()) {
    handle(f.get());  // crashes? what crashes?
}

• A`isDone()` is non-blocking, which is incorrect here.
• B`Future` cannot be used with lambdas.
• CThe executor should be a `ForkJoinPool`.
• DExceptions thrown inside `doWork()` are captured into the future and only surface when `get()` (or `exceptionally`/`handle`) is called — code that ignores those channels loses errors silently.

Multiple choice

A service depends on a third-party HTTP client that throws `OkHttpException`. The service's domain layer catches `OkHttpException` directly throughout the codebase. What's the chapter's recommended fix?

• AMock the client in production.
• BWrap third-party exceptions at the boundary into your own domain exception types, so the library's types don't bleed into callers and you can swap implementations.
• CCatch `Throwable` everywhere instead.
• DAdd `throws OkHttpException` to every domain method.

Multiple choice

What is the most accurate framing of checked vs. unchecked exceptions in a public library API?

• AAlways prefer checked — they're safer.
• BAlways prefer unchecked — they keep signatures clean.
• CUse checked for recoverable errors that are part of the contract; use unchecked for programmer errors and unrecoverable conditions.
• DUse only `RuntimeException` and document everything in Javadoc.

Balancing Flexibility and Complexity

The chapter examines tension between making an API extensible enough to accommodate diverse clients versus keeping it simple to maintain safely. Every public extension point becomes a permanent contract, so flexibility should be added deliberately rather than by default.

Start Simple, Add Flexibility on Demand

Begin with a robust but inextensible component. Introduce extension points only after identifying a concrete client need, avoiding speculative generality that costs forever.

Public Extension Points Are Forever

Removing functionality from a published API is a breaking change. Every hook or listener you expose becomes a maintenance liability you typically cannot retract.

Pluggable Frameworks via Dependency Injection

Rather than hardcoding a metrics library, let clients supply their own through an interface. This gives flexibility without ballooning the surface area of the core API.

Hooks Let Clients Inject Behavior, but Require Guards

A hooks API allows arbitrary client code to execute inside your component's lifecycle. You must defend against slow hooks, throwing hooks, and hooks that mutate shared state.

Performance Cost of Hooks Is Real

Every hook invocation adds overhead and places untrusted client code on your hot path. Even simple hooks can dominate runtime when called inside tight loops.

Listeners vs. Hooks

Listeners are notified after an event and cannot influence the host's behavior — invariants stay safe. Hooks are more powerful because they can alter outcomes, but that power is exactly what makes them dangerous.

Immutability Preserves Invariants When Extending

Designing listener payloads and hook inputs around immutable data prevents client extensions from corrupting internal state — a classic failure mode of plugin-style APIs.

Flexibility Cost-Benefit Analysis

For each proposed extension point, ask whether the gain in client capability justifies the cost of testing, documenting, and supporting it across future versions.

Extensibility

Ability for clients to add or modify behavior without forking the source.

Hook

A registered callback the host invokes at a defined point, able to alter inputs/outputs.

Listener

A registered callback notified of events but unable to change host behavior.

Pluggable framework

Design that accepts a client-supplied implementation behind an interface.

Immutability

Property of data objects that cannot be mutated after construction.

Breaking change

Any modification to a public API forcing clients to change their code.

API surface

Total set of types, methods, and configuration points a library exposes.

Spot the issue

A library author adds a `BeforeWriteHook` extension point "in case some future user wants to transform records." No client has asked for it. The hook is exposed as `public`, runs inside the write loop, and accepts mutable record objects. What's wrong?

• AHooks should never be added; only listeners.
• BHooks should always be `private`.
• C`BeforeWriteHook` is the wrong name.
• DThe author is speculatively adding a permanent public extension point, placing untrusted code on a hot path, and handing out mutable internal state — three flexibility costs without a concrete client need to justify them.

Multiple choice

A team needs to let clients observe record-write events without letting clients alter what gets written. Which extension mechanism best matches that intent?

• AA listener that fires after the write completes with an immutable payload.
• BA hook with mutable payloads, so clients can rewrite the record if they wish.
• CA subclass-only protected method.
• DA configuration string parsed at runtime.

Multiple choice

Why does the chapter recommend designing hook and listener payloads as immutable data?

• AImmutable objects are smaller in memory.
• BMutability would force a Java version upgrade.
• CClient extensions can otherwise corrupt internal state, which is a classic failure mode of plugin-style APIs.
• DImmutability is required by the JVM for callbacks.

Spot the issue

A logging library hardcodes Dropwizard Metrics as its sole metrics backend. A new client uses Micrometer and needs the library to emit metrics there instead. What's the better design?

• ADefine a small `MetricsSink` interface in the library and accept a client-supplied implementation via dependency injection — flexibility without ballooning the core API surface.
• BFork the library and rewrite it for Micrometer.
• CAdd `if (micrometer) { ... } else { ... }` branches throughout.
• DAlways emit to both backends simultaneously.

Premature Optimization vs. Optimizing the Hot Path

The chapter reframes Knuth's adage by distinguishing premature optimization (no data) from targeted optimization of the hot path (the small fraction handling most work). It demonstrates how to define SLAs, locate bottlenecks with load tests, and validate improvements with microbenchmarks.

Premature Optimization Needs No Data

Optimizing without an SLA or measurements typically wastes effort and complicates code. The same optimization, guided by data on a real hot path, can be the right call.

The 80/20 Hot Path

A small subset of routes or methods accounts for the vast majority of execution time. Concentrating effort on that subset yields disproportionate gains versus blanket optimization.

Define an SLA Before Optimizing

Throughput, latency, and concurrent-user targets convert "make it faster" into a falsifiable goal. You then know when to stop optimizing and avoid over-engineering.

Load-Test to Expose Hot Paths

Tools like Gatling configure concurrent users against your SLA, producing traffic realistic enough to surface bottlenecks that unit-level reasoning misses entirely.

Instrument With MetricRegistry

Adding timers and meters around suspected hot regions turns guesses into measurements, letting you locate the genuine hot path rather than the one you assumed existed.

JMH Microbenchmarks Validate Localized Changes

Once a hot method is identified, JMH provides JVM-aware, warmup-correct measurements that confirm a proposed optimization actually helps — and by how much.

Benchmark Before and After

A "performance improvement" may turn out to be no improvement at all. Always benchmark before and after rather than trusting intuition about what is slow.

Caching Has Its Own Tradeoffs

Caching dramatically improves throughput but introduces invalidation, memory footprint, and changing workload assumptions. The hit rate that justified it can shift the moment traffic patterns change.

Hot path

The subset of code executed for nearly every request; highest-leverage optimization target.

Premature optimization

Performance work undertaken before measurements justify it.

SLA

Specification of required throughput, latency, and capacity.

NFR

Nonfunctional requirement — how well the system performs vs. what it does.

Pareto principle

80/20 observation — a small portion produces most of the output.

Gatling

Scala-based load-testing framework for simulating concurrent users.

JMH

Java Microbenchmark Harness for JVM-aware microbenchmarks.

Microbenchmark

Small, focused performance test of a single method or fragment.

Spot the issue

An engineer says: "I'm rewriting the parser in hand-rolled bytecode-friendly Java because it's probably the hot path." There's no profile, no load test, no SLA — just a hunch. What's the mistake?

• AJava can't be hand-rolled.
• BParsers can't be optimized.
• CThe optimization is premature: with no SLA or measurements, the engineer is likely adding complexity to code that isn't on the real hot path.
• DThe team should use Kotlin instead.

Multiple choice

A service must serve 500 requests/second at p99 < 200ms for 1,000 concurrent users. Which artifact does the chapter say you should produce first, before any optimization work?

• AA written SLA capturing throughput, latency, and concurrency targets, so "make it faster" becomes a falsifiable goal.
• BA JMH benchmark of the suspected slow method.
• CA new caching layer.
• DA microservice extraction plan.

Multiple choice

Which tool combination does the chapter recommend for the workflow "find the real hot path, then validate a localized fix"?

• AJMH for both; it does load tests too.
• BOnly profilers; benchmarks aren't useful.
• CPrint statements and a stopwatch.
• DGatling for load-testing under SLA traffic; MetricRegistry timers/meters to locate hot regions; JMH for warmup-correct microbenchmarks on the candidate method.

Spot the issue

A team adds an in-memory cache fronting a query that runs 10,000x/minute, and ships immediately. Six months later, traffic patterns shift toward many unique queries, the hit rate drops below 5%, and the cache now consumes 8 GB of heap while doing little. What does the chapter say this team got wrong?

• ACaches should never be added.
• BCaching has its own tradeoffs — invalidation, memory footprint, and assumptions about workload — and the team didn't plan for the hit rate changing when traffic shifted.
• CThe cache should have been distributed from day one.
• DThe query should have been deleted.

Simplicity vs. Cost of Maintenance for Your API

The chapter compares two strategies for surfacing dependent-library configuration: directly re-exposing settings vs. wrapping them in a tool-specific abstraction. Through lifecycle events for adding and deprecating settings, it shows the abstracted API carries substantially higher long-term cost.

APIs Have UX, and UX Has a Price

REST endpoints, CLIs, and library configurations are user interfaces too. Making them ergonomic for callers requires sustained maintenance investment from the API owner.

Direct Exposure Minimizes Wrapper Maintenance

Forwarding the dependent client's configuration object means new underlying settings appear automatically — at the cost of leaking the dependent library's vocabulary to your users.

Abstracted Settings Improve UX, Demand Maintenance

Defining your own configuration types and translating to the underlying library gives users a cleaner, tool-shaped API. But every change in the underlying library must be re-mapped through the abstraction layer.

Configuration Is Part of the Public Contract

Users wire configuration into deployment scripts and infrastructure-as-code. Changing it is as breaking as changing method signatures, which constrains how either approach can evolve.

Asymmetric Cost of Adding Settings

The direct-exposure tool gets new settings for free. The abstracted tool requires a new field, validation, mapping code, documentation, and tests for every addition.

Asymmetric Cost of Removing Settings

Removing a setting from the direct tool forces an immediate user-facing break. The abstracted tool can keep the old name working via translation — trading short-term UX stability for long-term complexity.

Match Strategy to Evolution Rate

Stable libraries with sophisticated users tolerate direct exposure. Rapidly evolving dependencies with broad audiences justify the maintenance cost of abstraction.

API UX

How easy and pleasant an API is to learn, use correctly, and evolve with.

Direct exposure

Wrapper forwards a dependent library's configuration types unchanged.

Abstraction layer

Wrapper defines its own vocabulary and maps it onto the dependent library.

Deprecation

Marking a feature as discouraged and slated for removal, with a migration window.

Leaky abstraction

An abstraction that fails to fully hide its underlying implementation.

Configuration mechanism

The set of settings and validation rules parameterizing a library.

Multiple choice

A wrapper around a rapidly evolving SDK chooses to directly expose the SDK's configuration object. The SDK then adds a new connection-timeout setting. What is the natural consequence for the wrapper and its users?

• AThe setting becomes available to wrapper users automatically — for free — but at the cost of the SDK's vocabulary leaking through.
• BThe wrapper must release a new MAJOR version.
• CThe wrapper breaks until the maintainer adds the new field manually.
• DThe wrapper users lose the existing settings.

Spot the issue

A wrapper around an SDK chose the abstracted-settings approach, defining its own configuration types and mapping them to the underlying library. The SDK is on a 4-week release cadence and adds, renames, or removes settings every release. The wrapper team is now buried in mapping work and lags two SDK versions behind. What does the chapter suggest about the original strategy?

• AAbstraction is always wrong.
• BThe wrapper should switch languages.
• CThe SDK is at fault and should be replaced.
• DStrategy should match the dependency's evolution rate: abstraction's maintenance cost is hard to justify for a rapidly evolving library; direct exposure is often the better fit until the library stabilizes.

Multiple choice

Why does the chapter argue that configuration is part of the public contract?

• ABecause configuration files are stored in Git.
• BBecause configuration is internal and need not be versioned.
• CBecause users wire settings into deployment scripts and infrastructure-as-code, so changing setting names is as breaking as changing method signatures.
• DBecause YAML parsers are strict.

Spot the issue

A wrapper using the abstraction layer approach decides to remove a deprecated setting in the next release because the underlying SDK removed it. The maintainer skips the deprecation window. What does the chapter say is lost by skipping the abstraction's main advantage here?

• ANothing — removal is cheap either way.
• BThe whole point of the abstraction is that it can keep the old setting name working via translation, smoothing user migration; skipping deprecation surrenders that UX advantage and incurs the maintenance cost for no benefit.
• CThe wrapper should have used direct exposure all along.
• DRemoval requires a MAJOR version, regardless of approach.

Working Effectively With Date and Time Data

Largely authored by Jon Skeet (creator of Noda Time), this chapter argues date/time is deceptively simple and produces disproportionate production bugs. It builds precise vocabulary, then walks through scoping requirements, picking the right library, writing testable code, and surviving DST and tzdb edge cases.

Machine Time vs. Civil Time

Machine time is a single instant on the timeline; civil time is a wall-clock reading tied to a calendar and place. Mixing the two — storing a `LocalDateTime` when you need an `Instant` — is the most common date/time bug.

A Time Zone Is Not an Offset

A UTC offset (`+01:00`) is a fixed displacement. A time zone (`Europe/Warsaw`) is a rule set mapping instants to offsets over time. Storing only the offset loses DST information and breaks future calculations.

Limit Your Scope Before Writing Code

A calendar reminder needs civil time; a log timestamp needs an instant. Picking the narrowest representation avoids carrying complexity you'll never use.

Use a Modern Library

java.time (JSR-310) and Noda Time model the key distinctions at the type level. Legacy `java.util.Date`/`Calendar` and `System.DateTime` conflate concepts and force bugs at every boundary.

Inject a Clock for Testability

Calling `Instant.now()` directly hard-codes "the real clock" and makes tests flaky and time-bound. Pass a `Clock` abstraction so tests can supply a fixed or fake clock.

Serialize in ISO 8601 / RFC 3339

Use machine-parseable formats with explicit offsets (`2026-05-12T14:30:00Z`) when crossing processes or storage. Locale-dependent strings silently corrupt round-trips.

Calendar Arithmetic Is Not Commutative

Adding "1 month" then "1 day" need not equal "1 day" then "1 month" near month boundaries. Decide and document the policy rather than trusting library defaults blindly.

DST Creates Skipped and Ambiguous Times

Spring-forward skips local times; fall-back duplicates them. Code must explicitly choose a resolver strategy (push forward, push back, throw) — defaults vary by library and silently produce wrong instants.

Instant

A single unambiguous point on the global timeline, typically nanoseconds since epoch.

Epoch

The fixed reference point from which instants are measured (commonly Unix epoch).

LocalDateTime

Civil value with no zone or offset attached.

UTC offset

A fixed signed duration between local time and UTC at a specific moment.

Time zone (IANA / tzdb)

Named rule set mapping each instant to a UTC offset over time.

ZonedDateTime

Civil time combined with a zone, including full DST rules.

DST transition

Discontinuity producing skipped or ambiguous local times.

tzdb

IANA time zone database — the authoritative, frequently-updated dataset of zone rules.

Spot the issue

What's the mistake?

// Schedule a 9am wake-up alarm for tomorrow in the user's city
Instant alarm = Instant.now().plus(Duration.ofHours(24));
saveAlarm(alarm);

• A`Duration.ofHours(24)` is too long.
• B`Instant.now()` is not testable.
• CJava's `Duration` doesn't support hours.
• DA morning alarm is civil time anchored to the user's wall clock, not a fixed instant on the timeline; storing only an `Instant` will fire at the wrong wall-clock moment whenever DST changes between now and then.

Multiple choice

A service stores every event as `(local_datetime, utc_offset)` — for instance `(2026-05-12T14:30, +01:00)`. Six months later, the team needs to compute "this event time, but one year from now, in the same city." Why is this representation insufficient?

• AAn offset is a fixed displacement; a time zone is a rule set. Storing only the offset loses DST and future-rule information, so "same time in the same city" can't be reconstructed without the zone name.
• B`+01:00` is a deprecated format.
• C`LocalDateTime` is not Y2038-safe.
• DThe events should have been stored as strings.

Spot the issue

What's the mistake?

public class OrderService {
    public void place(Order o) {
        o.setPlacedAt(Instant.now());  // direct system clock
        repo.save(o);
    }
}

• A`Instant.now()` is too slow.
• B`placedAt` should be a `LocalDateTime`.
• C`Instant.now()` hard-codes the real clock, making tests time-bound and flaky; injecting a `Clock` lets tests supply a fixed or fake clock without changing production behavior.
• D`Instant` cannot be serialized.

Multiple choice

For data crossing process or storage boundaries, the chapter recommends serializing date/time values as:

• ALocale-aware formatted strings like `"5/12/2026 2:30 PM"`.
• BISO 8601 / RFC 3339 strings with explicit offset, e.g., `"2026-05-12T14:30:00Z"`.
• C`long` milliseconds without a unit comment.
• DWhatever the database driver picks by default.

Leveraging Data Locality and Memory of Your Machines

When datasets exceed a single machine's memory, the dominant cost is moving bytes over the network. This chapter explains data locality ("ship code to data"), contrasts in-memory vs. disk-based processing (Spark vs. MapReduce), and shows how to implement standard and broadcast joins.

Move Computation to Data

Serialized code is small; partitioned data is huge. Pushing the function to where the data already lives turns an O(dataset) network transfer into an O(code) one — the founding insight behind MapReduce and Spark.

Partitioning vs. Sharding

Partitioning splits one logical dataset within a system for parallel processing. Sharding splits it across independently administered systems. Mechanics overlap; operational implications differ.

Partitioning Algorithm Choice Matters

Hash partitioning gives uniform distribution but destroys range locality. Range partitioning preserves locality but invites hotspots. Choose based on which queries dominate.

Co-Partitioning Makes Joins Cheap

A join is cheap iff both sides are partitioned the same way on the join key. Otherwise the system must shuffle — a full network reshuffle that often dominates job time.

Broadcast Join for Small Sides

Sending a small lookup table to every executor lets each partition of the large table join locally. The tradeoff is memory pressure per executor and a hard upper bound on broadcast size.

MapReduce Disk-First vs. Spark RAM-First

MapReduce writes intermediate results to disk between every stage (resilient but slow). Spark keeps RDDs/DataFrames in memory across stages (fast but constrained by cluster RAM).

Memory Hierarchy Drives Design

RAM is five-to-six orders of magnitude faster than disk and three-to-four faster than network. Designs that minimize the slowest hop almost always win.

API Choice Encodes Join Strategy

Using `broadcast(df)` explicitly tells Spark to ship the small side. Without it, Spark may default to a sort-merge or shuffle-hash join that costs far more.

Data locality

Principle that computation should run on the node holding its input data.

Partition

One slice of a distributed dataset, assigned to a single executor.

Shard

Independently-managed partition, typically of a database.

Hash partitioning

Assigns rows by hash(key) mod N; uniform but locality-destroying.

Shuffle

All-to-all network exchange between executors, usually from a partition mismatch.

Broadcast join

Ships the small side to every executor, eliminating shuffle on the large side.

RDD / DataFrame

Spark's resilient distributed in-memory dataset abstractions.

Executor

A JVM process on a worker node that holds partitions and runs tasks.

Spot the issue

A Spark job joins a 2 TB `events` table with a 5 MB `country_codes` lookup table. The job runs for hours; the Spark UI shows a massive shuffle of `events` across the cluster. The code reads: What's the better join strategy?

events.join(country_codes, "country_id")

• ARepartition `events` by `id` and rerun.
• BConvert the job to MapReduce.
• CUse `events.join(broadcast(country_codes), "country_id")` so the tiny lookup is shipped to every executor and the huge side is joined locally — no shuffle on the 2 TB table.
• DIncrease executor count until the shuffle finishes faster.

Multiple choice

A Spark cluster is hash-partitioned by `user_id`. The team frequently runs range queries like "all events between user_id 1,000 and 2,000." Why is performance disappointing, and what's the tradeoff?

• AHash partitioning is always wrong; switch to range.
• BHash partitioning doesn't support user IDs.
• CThe cluster needs more RAM.
• DHash partitioning gives uniform distribution but destroys range locality; a range-query workload may justify range partitioning — accepting hotspot risk in exchange for locality.

Multiple choice

The data-locality principle "move computation to data" wins primarily because:

• ASerialized code is small, and shipping the function to the data turns an O(dataset) network transfer into an O(code) one.
• BCPU cycles are scarcer than network bandwidth.
• CModern networks are too unreliable for large transfers.
• DSpark requires it.

Spot the issue

A team picks Spark over MapReduce specifically because "RAM is faster than disk" — but their job's working set is 5x cluster memory, so Spark constantly spills to disk and the job underperforms a tuned MapReduce equivalent. What did they miss?

• ASpark is never appropriate.
• BSpark's RAM-first advantage holds only when the working set fits in cluster memory; MapReduce's disk-first design is resilient and predictable when it doesn't. The right engine depends on the workload's size vs. available RAM.
• CThey should have used Hive.
• DThe cluster needed an SSD upgrade.

Third-Party Libraries: Libraries You Use Become Your Code

Importing a library imports its behavior, bugs, concurrency model, license, and transitive dependencies — and you become responsible for all of them. The chapter gives a structured framework for choosing, configuring, testing, and maintaining dependencies, ending with an adopt-or-reimplement checklist.

Library Defaults Aren't Safe Defaults

Maintainers tune defaults for the common case across all users, rarely your case. Audit and override connection pool sizes, timeouts, retry counts, and serialization formats deliberately.

Sync vs. Async Locks In Concurrency Model

A blocking client called from a reactive pipeline eats your thread pool. An async client in synchronous code adds complexity for no benefit. Match the library's model to your application's.

Test With Fakes, Not Mocks of Third-Party Types

Mocking a library's classes couples your tests to its API surface and stubs in behavior the real code doesn't have. Prefer a vendor fake or a thin internal interface you own.

Use the Library's Integration Testing Toolkit

Many mature libraries ship a Testcontainers image, embedded server, or emulator. These give real behavior under test without production cost.

Transitive Dependencies Cause Dependency Hell

Two libraries may demand incompatible versions of a common third dependency. Resolve via build-tool pinning or dependency convergence — don't pretend the conflict isn't there.

Vendor Lock-In Is a Tradeoff, Not a Sin

A managed cloud SDK may give 10x productivity at the cost of portability. Make the choice explicitly and isolate the lock-in behind an interface if exit is plausible.

License Is a Property You Must Verify

Permissive licenses (MIT, Apache 2.0) carry few obligations. Copyleft licenses (GPL, AGPL) can require you to open-source your derived work. Getting this wrong means months of lawyer-driven rewrites.

A Library vs. a Framework

A library is a tool you call. A framework calls you — inversion of control. Frameworks give leverage but lock your application architecture to their lifecycle.

Transitive dependency

A library pulled in indirectly through another dependency.

Diamond dependency

Two dependencies demand incompatible versions of a third.

Test double

Generic term for any stand-in used in place of a real collaborator.

Fake

A working but simplified implementation suitable for tests.

Mock

A test double programmed with expectations and verified afterward.

Inversion of control

Framework calls your code rather than the other way around.

Vendor lock-in

Cost of switching away from a specific vendor or library.

Copyleft license

License requiring derivative works to be distributed under compatible terms.

Spot the issue

A SaaS company ships a closed-source product that links statically against a GPL-licensed library "because it had the best parser." Sales wants to keep the source private. What does the chapter say went wrong?

• ANothing — GPL applies only to direct copies.
• BStatic linking is always illegal.
• CThe library should have been re-implemented from scratch.
• DCopyleft licenses like GPL can require derivative works to be distributed under compatible terms; verifying license is part of choosing a dependency, and getting it wrong can mean lawyer-driven rewrites.

Multiple choice

Why does the chapter recommend vendor fakes or thin internal interfaces over Mockito-style mocks of third-party classes?

• AMocking a library's classes couples tests to its API surface and lets you stub behavior the real code doesn't have, so tests pass while production breaks.
• BMockito is deprecated.
• CMocks are slower than real instances.
• DMockito can't mock final classes.

Spot the issue

A service imports a popular HTTP client with default settings: unbounded connection pool, no read timeout, infinite retries. The service deploys to production and a single slow downstream causes thread starvation and a cascading outage. What's the chapter's framing?

• AThe HTTP client is buggy.
• BThe service should not use HTTP.
• CLibrary defaults are tuned for the common case across all users — not yours. Connection pool, timeouts, and retry counts must be audited and overridden deliberately.
• DUnbounded pools are always correct in production.

Multiple choice

A team is choosing between a managed cloud queue SDK (10x faster integration, deep cloud lock-in) and an open-source self-hosted broker (portable, more operational work). The chapter's framing is:

• AVendor lock-in is a sin; always pick the portable option.
• BVendor lock-in is a tradeoff, not a sin. Make the choice explicitly and, if exit is plausible, isolate the lock-in behind an interface.
• CAlways pick the managed service — productivity wins.
• DThe decision is purely a security one.

Consistency and Atomicity in Distributed Systems

Moving from a single-node application to a horizontally scaled deployment breaks assumptions about atomicity. This chapter walks through retries, idempotency, CQRS, and deduplication, showing how naive single-node implementations produce subtle race conditions when generalized.

At-Least-Once Is the Network's Default

When service A retries a call to service B, B may have already processed the request. The network cannot tell you which — designing around this reality is non-negotiable once you leave a single node.

Idempotency Makes Retries Safe

Producers must design operations so that applying the same request twice has the same effect as applying it once — upserts by stable business key rather than blind inserts. Otherwise retries silently corrupt state.

CQRS — Command Query Responsibility Segregation

Splitting the write path from the read path lets each side scale and evolve independently. The tradeoff is eventual consistency between the two models that callers must account for.

Naive Deduplication Breaks Under Concurrency

"Check if seen, then mark seen" is a classic check-then-act race. Two nodes processing the same message concurrently both pass the check before either mark lands.

Single-Node Solutions Hide Shared State

In-process maps or local caches give the illusion of correctness during development. They fail the moment the service is replicated.

Atomic Compound Operations Beat Locking

A single atomic primitive — conditional write, `INSERT ... ON CONFLICT`, compare-and-set — collapses check and act into one step and eliminates the race window.

Distributed Transactions Trade Availability for Consistency

Database transactions give atomicity for free on one node. Porting that guarantee across services requires two-phase commit or sagas at significant availability and latency cost.

Choose the Boundary of Atomicity Deliberately

Identify the smallest unit of work that must be atomic and design state so that unit fits inside a single storage primitive, rather than spreading it across services.

At-least-once delivery

Guarantee where a message is delivered one or more times; duplicates possible.

Idempotency

Performing an operation N times produces the same result as performing it once.

CQRS

Architectural pattern separating write-side commands from read-side query models.

Deduplication

Detecting and discarding duplicate requests, typically keyed on a request ID.

Race condition

Bug where correctness depends on unpredictable interleaving of concurrent operations.

Atomic operation

An operation executing as an indivisible unit; never observed in an intermediate state.

Distributed transaction

Transaction spanning multiple resource managers, coordinated via 2PC or sagas.

Spot the issue

What's the mistake?

// Deduplication across two app nodes sharing a database
if (!seen.contains(messageId)) {
    process(message);
    seen.add(messageId);
}

• A`seen` should be a `Set` not a `Map`.
• B`messageId` should be hashed first.
• C`process` should be wrapped in a try/catch.
• DThis is a check-then-act race: two nodes processing the same message concurrently both pass the `contains` check before either `add` lands. The fix is an atomic compound primitive — `INSERT ... ON CONFLICT DO NOTHING`, a conditional write, or compare-and-set — that collapses check and act into one step.

Multiple choice

A team migrates a single-node service to three replicas behind a load balancer. Their in-process dedupe map (`HashMap<UUID, Boolean>`) now silently lets duplicates through. The chapter's diagnosis is:

• AThe migration was premature.
• B`HashMap` is not thread-safe; `ConcurrentHashMap` would have fixed it.
• CIn-process state gave the illusion of correctness on one node; the moment the service was replicated, shared state had to move to a primitive both nodes could coordinate on.
• DThree replicas is the wrong number.

Multiple choice

Why does the chapter argue that idempotency is the foundation for safe retries in distributed systems?

• ABecause at-least-once is the network's default — the network can't tell A whether B already processed its prior request — so the producer must design operations so that applying them N times equals applying them once.
• BBecause retries are forbidden without it.
• CBecause idempotent code is faster.
• DBecause idempotency removes the need for timeouts.

Spot the issue

A microservices team needs an order-creation flow to atomically (1) charge a card via the payments service, (2) reserve inventory via the inventory service, and (3) persist the order in its own DB. They wrap all three in a single distributed transaction using two-phase commit. What's the tradeoff the chapter highlights?

• ATwo-phase commit is the only correct answer.
• BDistributed transactions across services trade availability and latency for consistency; choosing the smallest atomic unit and using sagas or idempotent compensations is often a better fit.
• CThe order service should handle payments itself.
• DInventory should never be reserved before payment.

Delivery Semantics in Distributed Systems

Building on Chapter 10, this chapter concretely shows how delivery semantics work in event-driven systems using Kafka as the running example. It walks through producer acks, idempotent and transactional producers, consumer offset management, and how combined choices yield at-most-once, at-least-once, or effectively exactly-once.

Delivery Semantics Are End-to-End

At-most-once, at-least-once, and exactly-once are emergent results of producer config, broker durability, and consumer commit timing combined. Tuning only one side is a common mistake.

Producer Acks Trade Durability for Latency

`acks=0` is fire-and-forget (fastest, can lose data). `acks=1` waits for the partition leader only. `acks=all` waits for all in-sync replicas — most durable, highest latency.

Idempotent Producers Eliminate Retry Duplicates

Kafka's idempotent producer attaches a producer ID and per-partition sequence numbers, letting the broker drop duplicate sends caused by retries on transient errors.

Transactional Producers Give Atomic Multi-Partition Writes

Wrapping sends and consumer offsets in a transaction lets a stream-processing job atomically commit what it read and what it wrote — the basis of Kafka's effectively-exactly-once.

Commit Timing Determines Semantics

Commit-before-process yields at-most-once (crash loses messages). Process-before-commit yields at-least-once (crash replays them). Auto-commit hides this choice and usually surprises you.

Exactly-Once Requires an Idempotent Sink

Kafka can deliver each record exactly once into Kafka itself. Writing to an external system (DB, HTTP API) only achieves exactly-once if that sink is idempotent or transactional.

Rebalancing Replays In-Flight Messages

When consumers join or leave the group, partitions are reassigned. Messages processed but not committed get redelivered to the new owner — where most at-least-once duplicates appear.

Offset Reset Shapes Recovery

Earliest replays all retained history (heavy, complete). Latest skips the backlog (light, accepts loss). Choose by workload — backfill vs. live feed.

At-most-once / at-least-once / exactly-once

Three canonical delivery guarantees.

Offset

Integer position of a record within a Kafka partition.

Consumer group

Set of consumers sharing the work of reading a topic.

Partition

Unit of parallelism and ordering in Kafka; records ordered within, not across.

In-sync replica (ISR)

Follower replica caught up with the leader, eligible for acknowledgment.

Replication factor

Number of copies of each partition maintained across brokers.

Idempotent producer

Kafka producer mode that deduplicates retries via producer ID + sequence numbers.

Rebalance

Process of redistributing partitions when consumer group membership changes.

Spot the issue

A Kafka consumer is configured with `enable.auto.commit=true` and runs: What's the delivery-semantics mistake?

for (ConsumerRecord<String, Order> r : poll()) {
    chargeCard(r.value());      // external HTTP call
    shipOrder(r.value());       // external HTTP call
}

• AAuto-commit hides commit timing. The offsets may commit before processing finishes — yielding at-most-once on crash — or after — yielding at-least-once with no idempotency at the sinks. The team should disable auto-commit, choose timing deliberately, and make the sinks idempotent.
• B`poll()` should be `pollAsync()`.
• C`chargeCard` should run before `poll`.
• D`ConsumerRecord` should be typed as `String`.

Multiple choice

A team claims their pipeline is "exactly-once" because they enabled Kafka's idempotent producer and use transactional reads. Downstream, the consumer writes results into PostgreSQL via plain `INSERT`. Where does the chapter say the guarantee actually breaks?

• AThe idempotent producer doesn't work with PostgreSQL.
• B`INSERT` is the wrong SQL verb.
• CTransactions don't span partitions.
• DKafka can deliver each record exactly-once into Kafka itself; writing to an external system only inherits exactly-once if the sink is idempotent or transactional. A blind `INSERT` is neither, so duplicates can land in Postgres on retry.

Multiple choice

Which producer `acks` setting trades the most durability for the lowest latency?

• A`acks=all`
• B`acks=1`
• C`acks=0`
• D`acks=isr`

Spot the issue

A consumer commits offsets every 30 seconds in a background thread. During a deployment, the consumer group rebalances, and a partition is reassigned to a different consumer mid-batch. The new owner replays the last 28 seconds of messages, causing duplicate charges downstream. What's the chapter's framing?

• ARebalances should be disabled.
• BRebalancing replays in-flight messages — processed but not committed — to the new owner. This is the primary source of at-least-once duplicates and must be paired with idempotent processing or tighter commit cadence.
• CThe deployment should not have happened during business hours.
• DKafka cannot recover from rebalances.

Managing Versioning and Compatibility

The chapter treats versioning as a first-class concern across four contexts: abstract version semantics, libraries, network APIs, and data storage. It distinguishes source/binary/semantic compatibility, explains diamond dependencies, and uses Protocol Buffers as the worked example for evolving persisted data.

Backward vs. Forward Compatibility

Backward-compatible means new code reads old data. Forward-compatible means old code tolerates new data. APIs and storage often need both, and they require different design moves.

Semantic Versioning Encodes a Contract

MAJOR.MINOR.PATCH communicates intent — MAJOR breaks, MINOR adds compatibly, PATCH fixes compatibly. The contract only works if you actually honor it, and marketing versions frequently don't.

Source, Binary, and Semantic Are Independent Axes

A change can recompile cleanly (source-compatible) yet break already-compiled callers (binary-incompatible), or vice versa. Semantic compatibility — same observable behavior — is a third orthogonal property.

Diamond Dependencies Force Single-Version Choice

When A depends on B v1 and C v2 of the same library, the build picks one. Library authors must preserve compatibility across MINOR/PATCH or risk wedging every downstream graph.

Internal Libraries Can Break Rules Deliberately

Inside one org with a monorepo or atomic deploys, you can co-evolve callers and library, removing much of the cost of breaking changes and shifting the tradeoffs.

Network APIs Need a Version-Discovery Story

URL-based (`/v1/...`), header-based, and content-negotiation strategies each trade clarity, caching, and routing. The chapter emphasizes customer-friendly clarity over cleverness.

Protocol Buffers Encode Evolution Rules

Adding fields with new tag numbers and never reusing or renumbering tags makes both backward and forward compatibility the default. Repurposing a tag number is the canonical breaking change.

Separate API Representation From Storage

Persisting your wire protobuf directly couples external API evolution to your database. A translation layer lets each version independently.

Semantic versioning

MAJOR.MINOR.PATCH scheme where each component signals a compatibility promise.

Backward compatibility

New producer/server works with old consumers/data.

Forward compatibility

Old consumer tolerates new producer/data it doesn't fully understand.

Source compatibility

Old caller code recompiles unchanged against the new library.

Binary compatibility

Old compiled artifacts keep linking and running without recompilation.

Diamond dependency

Build graph where two paths reach a library at different versions.

Protocol Buffers

Schema-driven binary serialization with tag-numbered fields enabling compatible evolution.

Breaking change

Any change violating the current compatibility contract.

Spot the issue

A team using Protocol Buffers needs to remove an obsolete field `string legacy_email = 4;`. They edit the `.proto` to delete the line and reuse tag number `4` for a new `int32 status_code = 4;` field. What's wrong?

• AProtobuf doesn't support `int32`.
• BThe field should be `int64`.
• CRenumbering or reusing a tag is the canonical breaking change in Protobuf — old data with tag 4 will deserialize into the wrong field type, silently corrupting both backward and forward compatibility. The fix is to reserve the old tag and assign the new field a fresh number.
• DField 4 should always be a string.

Multiple choice

A library author renames a `public` method's parameter type from a class to its newly-introduced interface and recompiles. Existing client source still compiles, but client artifacts compiled against the old JAR fail with `NoSuchMethodError` at runtime. Which compatibility axis was broken?

• ASemantic compatibility.
• BSource compatibility only.
• CNone — the rename is safe.
• DBinary compatibility — old compiled callers no longer link against the new artifact, even though source compiles cleanly.

Multiple choice

Service A depends on library L version 1.4. Service A also imports library M, which transitively depends on L version 2.0 (which dropped a method A uses). The build picks one version. What does the chapter say the library authors of L should have done?

• APreserved compatibility across MINOR/PATCH so neither downstream consumer was wedged by the choice the build tool was forced to make.
• BRelease as MAJOR every quarter.
• CStopped publishing.
• DRefused to support transitive use.

Spot the issue

A team persists its wire-format Protobuf messages directly into MongoDB as the storage schema. Two years later, the public API needs a breaking v2 redesign of one field. What does the chapter say is wrong with this setup?

• AMongoDB cannot store Protobuf.
• BPersisting the wire protobuf directly couples external API evolution to the database. A translation layer between API representation and storage representation lets each version independently — at the cost of an extra mapping layer the team must maintain.
• CThe team should switch to JSON.
• DProtobuf is unsuitable for persistence.