The Best Engineers Think in Trade-offs, Not Technologies

Part 1: The Search for the Best Technology

I used to ask a lot of questions that started with the word best.

What's the best database for this project?

What's the best backend framework?

Should we use REST or GraphQL?

Is MongoDB better than PostgreSQL?

Should this be a microservice?

What's the best way to deploy it?

These felt like sensible engineering questions. Somewhere out there, I assumed, there was a correct answer. My job was to research enough benchmarks, read enough documentation, compare enough architectures, and eventually find it.

The internet reinforced that belief.

Search for almost any technical decision and you'll find an endless supply of comparisons:

PostgreSQL vs MongoDB Kubernetes vs Docker REST vs GraphQL Monolith vs Microservices RabbitMQ vs Kafka

I've read plenty of them. Some were genuinely useful.

The problem wasn't the comparisons.

The problem was that I was looking for winners.

It took working on real systems for me to understand that engineering rarely gives you one.

The Database That Was Supposed to Scale

Imagine you're starting a new application.

Someone on the team says:

"We should use MongoDB. It'll scale better."

It's an attractive argument.

Nobody wants to be responsible for choosing the database that supposedly doesn't scale. So the discussion quickly moves toward MongoDB.

Then someone else asks:

"What kind of data are we storing?"

The room gets quieter.

That's a much less exciting question.

But it's the one that matters.

Are the relationships complicated?

Do we need transactions across multiple entities?

What queries will dominate the application?

How often does the schema change?

How much data are we realistically expecting?

Does the team already know how to operate MongoDB?

Suddenly, "Which database scales better?" feels like a strangely incomplete question.

Maybe MongoDB is the right choice.

Maybe PostgreSQL is.

The interesting part is that the answer has almost nothing to do with which database is better.

It depends on what you're building.

That sounds obvious when written down.

It took me longer than I'd like to admit to actually think that way.

Technologies Don't Exist in Isolation

A technology looks very different on its homepage than it does inside your production environment.

On the homepage, you see features.

In production, you inherit consequences.

Take Redis.

The first time you introduce caching into an application, it can feel almost magical. An expensive database query disappears behind a cache lookup and an endpoint that took hundreds of milliseconds suddenly responds almost instantly.

It's tempting to think:

Redis made the system faster.

Which is true.

But that's not the whole transaction.

You also introduced another system that can fail.

Now you need to decide how entries expire.

You need to think about invalidation.

You need to decide what happens when the cache and database disagree.

You need to monitor memory.

You need to answer a question that has ruined perfectly good afternoons for generations of engineers:

Why am I still seeing the old value?

You bought performance.

You paid with complexity.

That doesn't make caching a bad decision. I've used it many times and will continue to.

It just means the decision has a price.

Every Technical Decision Is a Purchase

I've started thinking about technical decisions almost like purchases.

When we introduce a technology, we're buying something.

Sometimes we're buying performance.

Sometimes reliability.

Sometimes developer productivity.

Sometimes scalability.

Sometimes flexibility.

But the purchase is never free.

The currency might be infrastructure cost.

Or operational complexity.

Or consistency.

Or latency.

Or engineering time.

Or simply the amount of information someone needs to hold in their head before they can safely modify the system.

The mistake I made earlier in my career was looking only at what I was buying.

I rarely looked closely enough at what I was paying.

Then You Discover Microservices

Microservices are probably the easiest example.

At some point, almost every backend engineer encounters an architecture diagram with dozens of neat little boxes.

User Service.

Payment Service.

Notification Service.

Order Service.

Authentication Service.

Everything has its own boundary.

Its own database.

Its own deployment.

The diagram looks clean.

Then you build one.

The first thing you discover is that splitting code is much easier than splitting a system.

A function call becomes a network request.

A database transaction becomes a distributed workflow.

Debugging one request may mean following it across several services.

Local development becomes more complicated.

Deployment requires coordination.

Observability becomes essential rather than optional.

And now a question that used to be:

"Did this function execute?"

becomes:

"Did Service A call Service B, did Service B publish the event, did Service C consume it, and if it didn't, where did the message go?"

Again, none of this means microservices are bad.

For some organizations, those costs are absolutely worth paying.

Independent teams can deploy independently.

Components can scale differently.

Failures can sometimes be isolated more effectively.

Large domains can be separated into understandable boundaries.

Those are valuable properties.

But you don't get them for free.

You exchange one kind of complexity for another.

The Monolith Wasn't Embarrassed

There's a strange moment that happens when you've spent enough time around distributed systems.

You start respecting boring architectures again.

A monolith stops looking primitive.

A relational database stops looking old-fashioned.

A cron job occasionally turns out to be exactly what you needed.

A simple HTTP endpoint begins looking surprisingly attractive compared with the event-driven workflow you were about to design.

This isn't regression.

It's context.

If five engineers are building an internal application for a few thousand users, perhaps operating twenty independently deployed services isn't sophistication.

Perhaps it's overhead.

If the application can comfortably run on a couple of instances, maybe Kubernetes isn't solving your most important problem.

If a database query takes 20 milliseconds, perhaps introducing a distributed cache isn't an optimization.

Maybe it's just another thing to maintain.

The difficult part is knowing when the boring solution stops being enough.

That's where engineering judgment begins to matter.

Scale Has a Context

"Will it scale?"

I've heard that question in architecture discussions more times than I can count.

It's a useful question, but only after another one:

Scale to what?

A system serving 500 users has different constraints from one serving 500,000.

An application receiving predictable business-hour traffic has different requirements from a ticketing platform that experiences sudden bursts.

A service handling financial transactions has different consistency requirements from an analytics dashboard.

We sometimes design for imaginary scale because imaginary scale is more interesting than actual requirements.

It's fun to discuss millions of requests per second.

It's less exciting to admit that the system might never receive more than fifty.

But architecture built for imaginary problems still creates real maintenance.

The Team Is Part of the Architecture

This is something technology comparisons almost always leave out.

Suppose two technologies can solve the same problem.

Technology A is theoretically more efficient.

Technology B is slightly less efficient, but everyone on the team understands it deeply.

Which one should you choose?

Earlier in my career, I probably would have picked A.

The technically superior choice.

Today I'd ask a lot more questions.

Who is going to operate it?

Who gets called when it breaks?

Can someone debug it at two in the morning?

How difficult will it be to hire people who understand it?

What happens when the engineer who introduced it leaves?

A technology can be excellent and still be wrong for a team.

That's uncomfortable because it means architecture isn't purely technical.

The people building and operating the system are part of the system.

Sometimes Performance Doesn't Matter

Engineers love performance.

Give us two implementations and we'll happily benchmark them.

One completes in 80 milliseconds.

The other takes 120.

Clearly, the first is better.

Except perhaps nobody cares.

If that operation runs once every evening as part of a background process, saving forty milliseconds may have absolutely no business value.

If the faster implementation is significantly harder to understand, we've optimized the wrong thing.

Performance matters when performance matters.

The same is true of scalability.

And flexibility.

And portability.

And almost every other "-ility" we put on architecture slides.

A property isn't valuable simply because it's technically desirable.

It's valuable when the system actually needs it.

"Best Practice" Needs a Sentence After It

I've become increasingly suspicious whenever I hear:

"That's best practice."

Best practice for what?

Under which constraints?

At what scale?

For which team?

With what failure model?

Most best practices began as solutions to real problems.

The trouble starts when we copy the solution without inheriting the problem.

Microservices solved organizational and scaling challenges at large technology companies.

Kubernetes solved orchestration problems for complex containerized workloads.

Event-driven architectures solve particular coupling and throughput problems.

None of that means a three-person team building a CRUD application needs all three.

Context is what turns a practice into a decision.

Without context, it's just imitation.

The Question Changed

I still learn new technologies.

I still enjoy trying new frameworks.

I still read architecture comparisons.

That hasn't changed.

What changed is what I'm looking for.

I no longer ask:

"Which one is better?"

I ask:

"What does each one make easier, and what does each one make harder?"

That's a much more useful question.

Because the interesting part of PostgreSQL isn't that it's better or worse than MongoDB.

The interesting part is what happens to your system when you choose it.

The interesting part of microservices isn't whether they're modern.

It's what you're accepting in exchange for independent boundaries.

The interesting part of Kubernetes isn't whether it scales.

It's whether the problems it solves are worth the operational system you're introducing.

Technology choices stop looking like rankings once you think this way.

They start looking like negotiations.

The Price Tag

I wish every technology came with a second documentation page.

The first page would contain what we already have:

Features

Fast
Scalable
Distributed
Flexible
Highly available

The second page would be called:

Things You're Going to Have to Deal With Later

That's the page I'd read first now.

Because almost every technology works beautifully in the happy path.

Engineering begins when you ask what happens outside it.

What happens when it fails?

What happens when traffic doubles?

What happens when the network is unreliable?

What happens when the data is inconsistent?

What happens when the person who designed it leaves?

What happens when you need to remove it?

Those questions aren't pessimism.

They're the price tag.

And learning to read that price tag has probably changed the way I make technical decisions more than any framework I've learned.

There is another side to trade-offs that took me even longer to understand.

When you choose one property, you're often quietly giving up another.

You choose consistency and sacrifice some availability.

You choose abstraction and sacrifice some transparency.

You choose flexibility and accept more complexity.

You choose speed of delivery and knowingly accumulate some debt.

Sometimes those are exactly the right decisions.

The important thing is recognizing that you made them.

Because architecture isn't about finding a solution with no disadvantages.

Those solutions don't exist.

It's about deciding which disadvantages you're willing to live with.

Part 2: Every Yes Is Also a No

There is a sentence I wish appeared more often in architecture meetings:

"What are we giving up?"

We are usually very good at discussing what a decision gives us.

Redis gives us faster reads.

Microservices give us independent deployment.

Queues give us asynchronous processing.

Abstractions give us flexibility.

Multi-region deployments give us availability.

Those statements aren't wrong.

They're just incomplete.

Every one of those decisions has another half.

Redis gives us faster reads, but now we have cache invalidation and another failure mode.

Microservices give us independent deployment, but turn local calls into network calls.

Queues decouple producers from consumers, but make execution harder to follow.

Abstractions give us flexibility, but can make simple behavior harder to see.

Multi-region infrastructure improves availability, but makes almost everything else more complicated.

The first half gets architecture diagrams.

The second half usually arrives six months later.

That's the part I've become more interested in.

The Feature Is Immediate. The Cost Is Delayed.

This is probably why overengineering is so easy.

Benefits tend to appear immediately.

Costs often don't.

Introduce a cache and the endpoint becomes faster today.

The stale-data bug appears three months later.

Split a service and the boundary looks beautifully clean today.

The distributed tracing problem appears when something fails in production.

Add an abstraction and the implementation looks elegant today.

Two years later, someone spends an afternoon jumping through six interfaces trying to understand what actually happens.

The distance between benefit and cost makes technical decisions deceptively cheap.

You get the thing you wanted immediately.

The invoice arrives later.

Sometimes to a completely different engineer.

Performance vs Understandability

Performance is one of the easiest places to see this.

Suppose you have an endpoint that takes 800 milliseconds.

You profile it and discover several expensive database queries.

There are many ways to improve it.

You could add caching.

Precompute the result.

Denormalize some data.

Introduce a read replica.

Maintain a materialized view.

Move part of the workload into a background process.

Any of those might work.

But each changes the shape of the system.

The question isn't only:

"How do we get this below 200 milliseconds?"

It's also:

"How much complexity are we willing to introduce to save those 600 milliseconds?"

Sometimes the answer is: quite a lot.

If it's the checkout endpoint of a high-volume commerce platform, those milliseconds may have measurable business value.

If it's an internal report opened twice a week, maybe 800 milliseconds is perfectly fine.

A technically slower system can sometimes be the better-engineered system.

That sentence would have bothered me earlier in my career.

It doesn't anymore.

Consistency vs Availability

Distributed systems make trade-offs impossible to hide.

Imagine a payment has just succeeded.

The payment service knows about it.

The order service hasn't received the event yet.

For a brief period, one part of the system says:

Payment: Successful

while another says:

Order: Pending

Is the system broken?

Maybe.

Maybe not.

If we've deliberately chosen eventual consistency, this could be expected behavior.

But then another question appears:

How much inconsistency can the business tolerate?

A few milliseconds?

Ten seconds?

Five minutes?

For an analytics dashboard, delayed data may be harmless.

For inventory during a limited sale, it might cause overselling.

For financial balances, it might be unacceptable.

"Eventual consistency" sounds like an architecture pattern.

In practice, it's a business negotiation about how long different parts of reality are allowed to disagree.

That's the part that matters.

Abstraction vs Transparency

Engineers are taught to avoid duplication.

It's generally good advice.

So we extract common behavior.

Then common behavior from that behavior.

Eventually we have something wonderfully reusable.

And nobody knows what it does.

I've worked with abstractions where understanding one operation required jumping through interfaces, base classes, configuration, dependency injection, and framework magic before finally reaching the code that actually performed the work.

The abstraction wasn't technically wrong.

It was simply more expensive to understand than the duplication it replaced.

That changed how I think about DRY.

Two similar pieces of code aren't automatically a problem.

Sometimes they're two things that happen to look similar today.

Combining them creates a relationship.

Relationships have maintenance costs.

Before removing duplication, I now want to know whether those things actually change for the same reason.

If they don't, a little duplication may be cheaper than the wrong abstraction.

Flexibility Is Not Free

Another word I've become suspicious of is flexible.

"We should make this flexible."

It sounds impossible to disagree with.

Who wants inflexible software?

But flexibility usually means supporting decisions that haven't been made yet.

Maybe we'll support multiple databases.

Maybe we'll add another cloud provider.

Maybe we'll need different authentication systems.

Maybe customers will want custom workflows.

So we design extension points.

Interfaces.

Configuration.

Adapters.

Factories.

Plugin systems.

Then the future arrives and none of those things happen.

We've spent real engineering time solving hypothetical problems.

Worse, every engineer now has to understand the flexibility we created.

I've started preferring a different strategy.

Build for what we know.

Leave enough room to change what we don't.

Those are not the same thing.

A system doesn't need to anticipate every future requirement.

It needs to be understandable enough that future engineers can modify it when those requirements actually arrive.

Speed vs Technical Debt

Technical debt is often discussed as though it's evidence of poor engineering.

I don't think that's fair.

Sometimes taking debt is exactly the right decision.

Imagine you're trying to validate whether anyone wants a product.

You could spend six months designing the perfect architecture.

Or you could build something imperfect in six weeks and discover whether the business should exist.

The second option may contain shortcuts.

Hardcoded assumptions.

Manual processes.

Limited scalability.

That's fine if they're deliberate.

The problem isn't technical debt.

The problem is forgetting that you borrowed something.

Debt becomes dangerous when a temporary shortcut quietly becomes permanent infrastructure.

"We'll fix this later" is reasonable.

"We'll fix this later" without an owner, a reason, or a trigger for when later arrives is how systems slowly become archaeological sites.

Sometimes speed is worth the debt.

Sometimes it isn't.

The engineering decision is knowing which situation you're in.

Reliability vs Cost

Ask whether a system should be highly available and almost everyone will say yes.

Then ask:

"How much are you willing to spend?"

The conversation becomes more useful.

Reliability has a price.

Redundant instances cost money.

Multi-zone deployments cost money.

Multi-region deployments cost considerably more money and engineering effort.

Backups cost money.

Disaster recovery environments cost money.

Operational readiness costs engineering time.

You can keep adding redundancy until almost every conceivable failure is covered.

At some point, the cost of preventing another minute of downtime becomes greater than the value of that minute.

That's uncomfortable because engineers naturally want systems to be reliable.

But "maximum reliability" isn't an architecture.

It's an unlimited budget.

Real systems need a target.

99% availability allows roughly 3.65 days of downtime per year.

99.9% allows about 8.76 hours.

99.99% brings that down to roughly 52.6 minutes.

Each nine becomes progressively more expensive.

The question isn't:

"Can we make this more reliable?"

We almost always can.

The question is:

"What level of reliability does this system justify?"

A payroll system and an internal lunch-menu application probably shouldn't have the same availability architecture.

That isn't negligence.

It's prioritization.

Build vs Buy

Engineers like building things.

That's partly why we became engineers.

Which makes "build or buy?" surprisingly difficult to answer objectively.

Building gives you control.

You can shape the system around your exact requirements.

You own the roadmap.

You avoid some vendor constraints.

It also means you own everything else.

Security patches.

Upgrades.

Availability.

Documentation.

On-call.

Edge cases.

The obscure bug that happens only on the last Friday of February.

Buying transfers some of those problems elsewhere.

But then you accept a different set.

Vendor dependency.

Pricing changes.

API limitations.

Data portability.

Roadmap decisions you don't control.

Again, there is no winner.

You're choosing which problems you'd rather own.

I've come to think that's the heart of the build-vs-buy decision.

Not:

"Can we build it?"

Engineers can build almost anything given enough time.

The better question is:

"Is this a problem we want to be responsible for?"

Portability vs Using the Platform Properly

I've seen architectures designed around the requirement:

"We shouldn't be locked into a cloud provider."

Reasonable.

So every cloud-specific capability gets hidden behind an abstraction.

Managed services are avoided.

The application uses only capabilities that exist everywhere.

And eventually the system is portable.

It can run equally awkwardly on three clouds.

Portability has value.

Sometimes enormous value.

But portability also has a cost.

If avoiding lock-in means refusing every useful managed service, you may end up operating infrastructure your cloud provider could have managed for you.

The real question isn't whether vendor lock-in exists.

It always exists somewhere.

Your programming language is a dependency.

Your database is a dependency.

Your framework is a dependency.

Your team's expertise is a dependency.

The useful question is whether the dependency is acceptable relative to the value it provides.

Sometimes lock-in is a terrible trade.

Sometimes it's the reason a small team can operate a system at all.

Synchronous vs Asynchronous

Suppose Service A needs Service B to perform some work.

The simplest version is often:

Service A
    |
    | HTTP request
    v
Service B

Easy to understand.

Easy to debug.

And Service A now depends on Service B being available.

So someone suggests a queue.

Service A
    |
    | event
    v
   Queue
    |
    v
Service B

Now Service A can continue even if Service B is temporarily unavailable.

We've gained resilience and decoupling.

We've also changed the programming model.

When does the work complete?

What happens if the message is delivered twice?

What happens if processing fails after partially completing?

How do we replay failed events?

How does the user know the operation eventually succeeded?

Where do we look when something disappears between producer and consumer?

Asynchronous systems solve real problems.

They also create real problems.

The important thing is making sure you're exchanging one for the other intentionally.

Security vs Convenience

Some trade-offs aren't symmetrical.

Security deserves special treatment because the easiest system to use is often the easiest system to misuse.

Give a service broad permissions and development becomes easier.

Until those permissions are exploited.

Remove authentication from an internal endpoint and integration becomes simpler.

Until "internal" stops meaning what you thought it meant.

Store a credential somewhere convenient and everyone moves faster.

Until it leaks.

Security often introduces friction deliberately.

The mistake is treating all friction as bad engineering.

Sometimes friction is the feature.

The challenge is putting it in the right place.

A secure system that nobody can use will eventually be bypassed.

A convenient system with no meaningful controls will eventually hurt you.

Good security architecture doesn't maximize restriction.

It makes the safe path easier than the unsafe one.

Observability vs Everything You Can Possibly Log

I used to think more logs meant better observability.

Then I had to search through them.

There is a point where more telemetry creates less visibility.

If every function logs its entry and exit, every request produces thousands of lines, and every metric gets its own dashboard, technically you can observe almost everything.

Practically, finding the one thing that matters during an incident becomes harder.

Observability has its own signal-to-noise problem.

What do we need to know?

What will help us diagnose failure?

Which business events matter?

Which metrics actually indicate health?

Collecting data is easy.

Knowing what deserves attention is the engineering work.

Simplicity vs Flexibility

This may be the trade-off I think about most often.

Simple systems are wonderful.

Until they need to change.

Flexible systems adapt easily.

Until nobody understands them.

There's no permanent answer.

A system that is too rigid becomes expensive to evolve.

A system designed for every imaginable future becomes expensive before that future even arrives.

The balance I've come to prefer is simple:

Make today's requirements obvious.

Make tomorrow's changes possible.

Don't try to implement tomorrow today.

That sounds straightforward.

In practice, it requires restraint.

Sometimes the Right Decision Looks Technically Worse

This is perhaps the part of engineering maturity I appreciate most now.

The right decision can look unimpressive.

Use the database you already operate.

Keep the monolith.

Accept the 300-millisecond response.

Run the scheduled job once an hour.

Use the managed service.

Duplicate twenty lines of code.

Ship the imperfect version.

Don't introduce the framework.

These decisions don't make exciting conference talks.

They don't produce beautiful architecture diagrams.

Sometimes they're exactly right.

Because architecture isn't a competition to build the most sophisticated system.

It's an attempt to build the least complicated system that satisfies the constraints you actually have.

There Is No Free Architecture

Every architecture has a bill.

Sometimes you pay in money.

Sometimes latency.

Sometimes consistency.

Sometimes operational complexity.

Sometimes developer experience.

Sometimes future flexibility.

The dangerous architectures aren't necessarily the expensive ones.

They're the ones where nobody knows what they're paying for.

That's why I increasingly want trade-offs written down.

Not just:

"We chose Kafka."

But:

"We chose Kafka because we need durable asynchronous processing and multiple independent consumers. We accept the additional operational complexity because those properties matter to this workload."

Now there's a decision.

More importantly, there's something a future engineer can challenge.

If those assumptions stop being true, perhaps the technology should disappear too.

Architecture Decisions Have Expiry Dates

This is another thing I wish I'd understood earlier.

A good decision can become a bad decision without anyone making a mistake.

The system changes.

Traffic changes.

The team changes.

The business changes.

Cloud services improve.

Costs change.

A database selected five years ago may have been exactly right at the time and completely wrong today.

That doesn't mean the original architect failed.

It means architecture has context.

And context has a habit of changing.

This is why "we've always done it this way" is such a dangerous sentence.

The original trade-off may no longer exist.

Every Yes Is Also a No

Choose microservices and you're saying yes to independent boundaries.

You're also saying no to some of the simplicity of local calls and transactions.

Choose eventual consistency and you're saying yes to availability and decoupling.

You're also saying no to the assumption that every part of the system always agrees immediately.

Choose abstraction and you're saying yes to flexibility.

You're also giving up some directness.

Choose maximum reliability and you're saying yes to redundancy.

You're also saying yes to paying for it.

Choose speed today and you may be borrowing maintainability from tomorrow.

That's engineering.

Not finding the option with no downside.

Understanding the downside well enough to decide whether it's worth accepting.

What Changed for Me

I still have technology preferences.

Everyone does.

There are databases I enjoy working with.

Languages I reach for first.

Architectures I understand better than others.

But preferences are no longer enough to justify a decision.

I've become much more comfortable saying:

"It depends."

Not as an escape from answering.

As the beginning of answering properly.

Depends on traffic.

Depends on the team.

Depends on the failure model.

Depends on the budget.

Depends on how quickly this needs to ship.

Depends on how long we expect it to live.

Once those things are understood, the technology decision often becomes surprisingly boring.

And boring is frequently a good sign.

The longer I work in engineering, the less I think our job is to eliminate problems.

Every system has problems.

Our job is to choose them.

A monolith has problems.

Microservices have different problems.

Running your own infrastructure has problems.

Managed services have different problems.

Strong consistency has problems.

Eventual consistency has different problems.

There is no architecture on the other side of enough experience where all of these tensions disappear.

Experience simply changes which questions you ask before accepting them.

And that leads to the final part of this essay:

Maybe engineering judgment isn't knowing the right solution. Maybe it's knowing which problems you're willing to live with.

Part 3: Choosing the Problems You Want

For a long time, I thought the goal of architecture was to solve problems.

Choose the right database.

Design the right API.

Find the right deployment model.

Build the right abstractions.

If you made enough good decisions, I assumed you'd eventually arrive at a system where most of the difficult problems had disappeared.

I don't believe that anymore.

Every architecture I've worked with has problems.

The good ones aren't problem-free.

They simply have problems that make sense for what they're trying to do.

That distinction has changed the way I think about engineering.

You Don't Choose Technologies. You Choose Problems.

Suppose you're deciding between a monolith and microservices.

With a monolith, you may eventually deal with:

Growing codebase complexity
Deployment coordination
Shared database boundaries
Increasing coupling between modules

With microservices, you may deal with:

Network failures
Distributed tracing
Eventual consistency
Service discovery
Deployment orchestration
Cross-service debugging

Neither option removes complexity.

It moves it.

The real question isn't:

"Which architecture is better?"

It's:

"Which set of problems makes more sense for this system and this team?"

That framing is far less exciting.

It's also far more useful.

The Problems You Know Are Often Cheaper

Engineers naturally enjoy new technology.

I certainly do.

There's something satisfying about learning a new database, framework, runtime, or infrastructure tool.

But novelty has a hidden cost.

Suppose your team has operated PostgreSQL successfully for five years.

Everyone understands it.

Backups are automated.

Monitoring exists.

People know how to investigate slow queries.

The failure modes are familiar.

Then a new project arrives.

Someone suggests introducing another database because one part of the workload fits it slightly better.

Maybe they're right.

But the comparison isn't:

PostgreSQL
vs
New Database

It's closer to:

PostgreSQL
+ five years of operational knowledge

vs

New Database
+ everything we haven't learned yet

That second term matters.

Operational familiarity is an architectural asset.

It rarely appears on benchmark charts.

Familiarity Isn't the Same as Complacency

There is a danger here.

"We already know this" can easily become an excuse never to improve.

That's not what I'm arguing for.

Sometimes the existing technology genuinely becomes the wrong choice.

Sometimes the benefits of changing are large enough to justify the migration.

Sometimes a new capability changes what's possible.

The important part is recognizing the full cost.

Introducing a technology doesn't end when the pull request is merged.

Someone has to operate it.

Upgrade it.

Secure it.

Debug it.

Teach it.

Eventually replace it.

The adoption cost is only the beginning.

Reversibility Changed How I Make Decisions

Not every architectural decision deserves the same amount of debate.

This sounds obvious, but I've sat in meetings where teams spent hours discussing decisions that could have been changed the next afternoon.

I've also seen decisions with years of consequences made in fifteen minutes.

A useful question is:

How expensive will this be to reverse?

Choosing the structure of an internal module?

Probably reversible.

Choosing the primary data model for a system expected to live for a decade?

Less so.

Naming an internal API endpoint?

Easy to change.

Publishing that API to hundreds of external customers?

Very different.

Selecting a small library?

Usually recoverable.

Building your entire platform around a proprietary capability?

Worth considerably more thought.

The harder a decision is to reverse, the more carefully I want to understand its trade-offs.

Not every choice needs an architecture review.

Some absolutely do.

Optionality Has Value

Sometimes the best architecture isn't the one optimized perfectly for today.

It's the one that leaves reasonable options tomorrow.

That doesn't mean building abstractions for every imaginary future.

It means avoiding decisions that unnecessarily close doors.

There's a difference.

Suppose you're unsure whether a workload will eventually need to move into its own service.

You don't necessarily need to create that service today.

But keeping the module boundary clean may make the extraction easier later.

That's useful optionality.

Compare that with designing an elaborate plugin framework because:

"We might need plugins someday."

That's speculation.

Good architecture doesn't predict the future.

It avoids making the future unnecessarily expensive.

Some Decisions Should Be Intentionally Temporary

Engineers don't like temporary solutions.

Mostly because we've learned what "temporary" usually means.

Five years later, somebody finds a comment:

// Temporary workaround. Remove after migration.

Nobody remembers the migration.

The workaround now processes half the company's traffic.

But temporary solutions aren't inherently bad.

Sometimes they're exactly what you need.

The difference is whether the temporary nature exists only in someone's head or is encoded into the way the team works.

If we're taking a shortcut, I want to know:

Why are we taking it?

What are we gaining?

What breaks if it stays?

When should we revisit it?

What event tells us the trade-off is no longer acceptable?

"Until we have time" isn't a trigger.

"When traffic exceeds X" is.

"When the second customer needs this behavior" is.

"When maintaining this manually takes more than two hours per week" is.

Temporary architecture works much better when it has an expiry condition.

Good Engineering Sometimes Means Doing Nothing

This took me a while to learn.

Not every problem needs to be solved immediately.

A database query is slightly slow.

A service isn't perfectly separated.

Two pieces of code are duplicated.

An internal tool occasionally requires a manual step.

The engineering instinct is to fix them.

But fixing something has a cost too.

You have to understand the problem.

Design the change.

Implement it.

Test it.

Deploy it.

Monitor it.

And accept whatever new behavior the fix introduces.

Sometimes the existing imperfection costs less than the solution.

That's not laziness.

It's prioritization.

A system can contain known imperfections and still be well engineered.

The important thing is knowing which imperfections are acceptable.

The Cheapest Incident Is Sometimes the One You Accept

Reliability engineering makes this particularly clear.

Imagine an internal reporting system fails once every few months.

Restarting one process fixes it.

The interruption lasts five minutes.

You investigate and discover you could eliminate the failure entirely.

It would require redesigning part of the system, adding redundancy, and spending several weeks implementing and validating the change.

Should you?

Maybe.

But "we know how to fix it" isn't enough.

How expensive is the incident?

How often does it happen?

Who is affected?

What's the engineering cost of eliminating it?

What higher-value work won't happen because the team is fixing this?

Engineers naturally dislike known failure modes.

Businesses sometimes rationally accept them.

Architecture lives in the uncomfortable space between those two perspectives.

Reliability Is a Business Decision Too

This is why I don't think engineers can make good architecture decisions without understanding the business.

Suppose someone says:

"This service cannot go down."

That isn't a requirement yet.

What happens if it does?

Do customers lose money?

Does production stop?

Can requests wait?

Can users retry?

Does another system take over?

How long can the outage last before it becomes serious?

The answers determine architecture.

A service that genuinely cannot tolerate downtime might justify multi-region failover, redundant dependencies, aggressive monitoring, and expensive operational practices.

A dashboard used by twelve employees probably doesn't.

Both deserve good engineering.

They don't deserve the same engineering.

Constraints Aren't Annoyances

Earlier in my career, constraints felt like things preventing us from building the ideal solution.

Not enough time.

Not enough budget.

Not enough people.

Legacy systems.

Compliance requirements.

Existing infrastructure.

Now I think those constraints are the architecture problem.

Without constraints, almost any system is easy to design.

Give me unlimited money, unlimited engineers, unlimited time, and no legacy dependencies, and I can draw you a wonderful architecture.

It will also be completely useless as an engineering exercise.

The interesting decisions happen because resources are finite.

You have three engineers.

Six weeks.

An existing database you cannot replace.

A regulatory requirement you cannot negotiate.

A traffic spike every Friday.

Now we have architecture.

Constraints don't prevent engineering.

They create it.

Sometimes You Pay Money to Remove Engineering

There's another trade-off engineers occasionally underestimate.

We can spend engineering effort to save money.

Or spend money to save engineering effort.

Managed services make this obvious.

Running your own database may be cheaper on an infrastructure invoice.

But infrastructure isn't the entire cost.

Who handles backups?

Replication?

Failover?

Patching?

Monitoring?

Recovery?

If a managed database costs significantly more but removes hundreds of hours of operational work, the "expensive" option may actually be cheaper.

The reverse can also be true at sufficient scale.

That's why cost optimization can't stop at the cloud bill.

Engineer time is infrastructure too.

It's just much harder to graph.

Organizational Architecture Is Still Architecture

Suppose you design a beautifully decoupled microservice architecture.

There are ten services.

Each has clean ownership boundaries.

Then one team owns all ten.

Have you actually gained independent service ownership?

Technically, perhaps.

Organizationally, not really.

Or imagine the opposite.

Five teams work inside one enormous codebase where every change requires coordination with everyone else.

The technical architecture is now shaping the organization.

This is why architecture diagrams that contain only software components always feel incomplete to me.

There are people behind every box.

Teams.

Ownership.

On-call rotations.

Knowledge.

Communication paths.

If the architecture ignores those things, reality eventually adds them back in.

Usually during an incident.

The Best Decision May Change as the Team Changes

A three-person startup and a 300-engineer organization can look at the same technical problem and reasonably choose completely different architectures.

That's not inconsistency.

It's context.

At three engineers, a monolith may allow everyone to move quickly.

At 300, clear service boundaries may allow teams to work without constantly coordinating.

Then again, badly chosen service boundaries can make 300 engineers coordinate even more.

Team topology matters.

Operational maturity matters.

Experience matters.

Architecture that ignores the people operating it is architecture for an imaginary organization.

Good Architecture Gives Failure Somewhere to Go

I've started paying much more attention to what systems do when something goes wrong.

Not:

"Can this fail?"

Everything can.

Instead:

"Where does the failure go?"

If an email provider is unavailable, does checkout fail?

If analytics is down, does the user-facing application stop working?

If one consumer cannot process an event, does it block every other message?

If a cache disappears, does the database collapse under the fallback traffic?

Failure boundaries tell you a lot about architecture.

Good systems don't prevent every failure.

They stop failures from becoming larger than they need to be.

That's a much more realistic goal.

Graceful Degradation Is Often Better Than Perfection

Imagine a commerce application.

Recommendations fail.

Should customers still be able to buy products?

Of course.

Search analytics fail.

Should search stop working?

Probably not.

A profile image service becomes unavailable.

Should authentication fail?

Hopefully not.

Yet systems accidentally create these dependencies all the time.

A non-critical component becomes part of a critical path because it was convenient to call synchronously.

Then one day the recommendation engine takes down checkout.

Architecture isn't just deciding how components communicate.

It's deciding which components are allowed to hurt each other.

The Trade-off Nobody Talks About: Cleverness vs Debuggability

I've written code I was proud of because it was clever.

I've also debugged clever code written by someone else.

Those experiences changed my priorities.

Clever systems often optimize for the moment they're written.

Debuggable systems optimize for the moment they're broken.

And every sufficiently important system eventually breaks.

When that happens, I want boring things.

Clear logs.

Obvious control flow.

Predictable behavior.

Few hidden abstractions.

Errors that explain what actually happened.

The architecture that looks slightly less elegant on a whiteboard may be dramatically easier to operate at 3 a.m.

That matters more to me now than it used to.

Design for the Incident, Not Just the Demo

Demos are dangerous.

Everything works.

The database responds.

The API is available.

The queue is empty.

The network behaves.

Production eventually introduces reality.

Timeouts.

Duplicate messages.

Partial failures.

Slow dependencies.

Expired credentials.

Full disks.

Unexpected traffic.

Bad deployments.

Humans making mistakes.

One of the questions I now like asking during design is:

"Tell me how this fails."

Not because I expect pessimism.

Because the failure story often exposes the architecture more clearly than the success story.

If nobody can explain what happens when a dependency disappears, we probably don't understand the system yet.

You Are Always Borrowing From Somewhere

Most architecture decisions feel like moving resources between accounts.

Ship faster today?

You may borrow maintainability from next quarter.

Add abstraction now?

You spend cognitive load to buy flexibility.

Denormalize the data?

You buy read performance with consistency complexity.

Add redundancy?

You buy reliability with money and operational overhead.

Use a managed platform?

You buy engineering time with vendor dependency.

There's nothing inherently wrong with any of those exchanges.

Problems appear when we pretend the exchange didn't happen.

Technical debt is manageable when you know you borrowed.

Complexity is manageable when you know what it purchased.

Vendor lock-in is manageable when the value exceeds the switching risk.

The trade-off isn't the problem.

Unexamined trade-offs are.

Architecture Is Mostly About Consequences

The longer I've worked with software, the less impressed I am by architecture diagrams alone.

You can draw almost anything.

A beautifully symmetrical diagram tells me very little about whether the system is good.

I want to know what happens next.

What happens when traffic grows?

What happens when a service fails?

What happens when the team doubles?

What happens when the schema changes?

What happens when the vendor raises prices?

What happens when the original engineers leave?

Architecture isn't the arrangement of boxes.

It's the consequences created by their arrangement.

Knowing When to Revisit a Decision

If architecture decisions have context, and context changes, then we need to be willing to revisit them.

But continuously reconsidering every decision would be exhausting.

So I like thresholds.

We keep the monolith until deployment coordination becomes a measurable bottleneck.

We keep the simple database until its limitations actually affect the workload.

We keep the manual process until its operational cost becomes meaningful.

We don't migrate because something newer exists.

We migrate because an assumption behind the original decision stopped being true.

That's a much healthier reason to change architecture.

"It Depends" Is Not the Answer

Experienced engineers have a reputation for answering everything with:

"It depends."

Sometimes that's wisdom.

Sometimes it's avoidance.

The useful part comes after it.

It depends on what?

That's where engineering begins.

If someone asks whether we should use a queue, "it depends" isn't useful.

But this might be:

"It depends on whether the caller needs the result immediately, how we want failures to behave, whether duplicate processing is acceptable, and how much operational complexity we're willing to introduce."

Now we've identified the variables.

The technology choice can follow.

"It depends" should open the discussion.

Not end it.

The Engineer I Trust

If two engineers are discussing a new architecture, I'm increasingly less interested in the one who can name the most technologies.

I'm interested in the one asking questions like:

What happens when this fails?

Why does this need to be distributed?

Can we remove this component?

What assumption are we making here?

How difficult is this to reverse?

Who operates this?

What's the simplest version that satisfies the requirement?

What are we paying for this property?

Those questions don't sound particularly advanced.

That's probably why they're useful.

They force the conversation away from technology and toward consequences.

Experience Doesn't Eliminate Mistakes

I don't want to romanticize experience.

Senior engineers make bad architecture decisions too.

Architects overengineer systems.

Experienced teams choose fashionable technologies for weak reasons.

I've certainly made decisions I'd approach differently today.

Experience doesn't give you a perfect answer.

What it can give you is a better instinct for where the problems are hiding.

You start noticing costs that aren't visible yet.

You recognize familiar failure modes.

You know which "temporary" decisions tend to become permanent.

You become suspicious of flexibility nobody asked for.

You ask who will operate the thing after everyone stops being excited about building it.

Maybe that's engineering judgment.

Not predicting the future.

Just becoming slightly harder to surprise.

Choosing the Problems You Want

This is where I've ended up.

I no longer believe architecture is primarily about choosing the right technologies.

It's about choosing the right problems.

Use a monolith and accept its constraints.

Use microservices and accept theirs.

Choose a managed service and accept dependency on the provider.

Run it yourself and accept operational ownership.

Optimize for performance and accept some complexity.

Optimize for simplicity and accept that you may need to redesign something later.

Every direction contains inconvenience.

The goal isn't to escape it.

The goal is to make sure the inconvenience you've chosen is aligned with what actually matters.

The Best Technology Is Usually a Boring Conclusion

After enough discussion about requirements, constraints, failure modes, team capabilities, reversibility, and cost, something funny happens.

The technology choice often becomes obvious.

And usually not very exciting.

Use PostgreSQL.

Keep the existing service.

Add a queue here.

Don't add one there.

Use the cloud provider's managed database.

Leave this synchronous.

Cache this one expensive path.

Don't build the abstraction yet.

The architecture meeting becomes less about discovering a clever solution and more about agreeing on which compromises make sense.

I think that's a good outcome.

Technology should often be the conclusion of an architecture discussion.

Not the beginning.

Final Thoughts

I used to think becoming a better engineer meant accumulating answers.

Learn more languages.

Know more databases.

Understand more frameworks.

Study more architecture patterns.

Those things helped.

But somewhere along the way, the bigger change was learning to ask different questions.

Not:

What's the best database?

But:

What behavior does our data actually require?

Not:

Should we use microservices?

But:

What problem would distributing this system solve?

Not:

Can this scale?

But:

What scale do we actually need, and what are we willing to pay for it?

Not:

Is this best practice?

But:

Which assumption made this practice useful, and does that assumption apply here?

Those questions don't produce universal answers.

They're not supposed to.

Engineering happens inside constraints.

Different constraints produce different good decisions.

That's why copying another company's architecture rarely gives you their results. You inherit their solution without necessarily inheriting the problem that justified it.

The technologies matter.

Of course they do.

But technologies are tools, and tools only make sense in relation to the work being done.

The part that takes longer to learn is understanding the exchange underneath every choice.

What are we gaining?

What are we giving up?

Who pays for the complexity?

When does this decision stop making sense?

How difficult will it be to change?

If you can answer those questions, choosing the technology is often the easy part.

Maybe that's why the engineers I respect most don't seem particularly attached to technologies.

They'll use a monolith when a monolith makes sense.

They'll distribute a system when the constraints justify it.

They'll choose the boring database.

They'll pay for the managed service.

They'll write the simple cron job.

And when the context changes, they'll change their mind.

Not because the old decision was wrong.

Because engineering isn't about being loyal to decisions.

It's about being honest about their trade-offs.

I used to think experience meant knowing more technologies.

Now I think it means needing fewer of them to prove that you know what you're doing.

The technology is rarely the difficult part.

Understanding what you're willing to live with is.