SOCRATES: Two scrolls and neither one open. Either both are bad, or you do not know which to read first.

GLAUCON: Neither, Socrates. They are too good, and that is the problem. This one is by Damon, with whom I have studied for three years. That one is by Theophilus, whom I met last winter. I have never heard anyone explain things the way those two do.

SOCRATES: And what is the affliction in having two good masters?

GLAUCON: Yesterday I quoted Theophilus in front of Damon. He went silent, then said Theophilus is a charlatan and that I would be wasting my time. Today I told Theophilus, and he laughed and said Damon is an old man repeating what he read at twenty. Each of them is everything I admire. And each despises the other.

SOCRATES: And what do you conclude from that?

GLAUCON: That one of the two is wrong, and that I need to find out which, so I can follow the right one.

SOCRATES: Tell me something first. Why did you look for a master?

GLAUCON: Because one does not learn alone. Whoever has already walked the road spares the others from tripping on the same stones.

SOCRATES: That is true, and it is good that you looked. But answer honestly: when you found Damon, was it only the road that interested you, or also the rest?

GLAUCON: What rest?

SOCRATES: The rest of no longer having to decide. Having someone to ask “is this right?” and accepting the answer. Sleeping in the knowledge that, if you err, the error belongs to whoever taught you.

GLAUCON: I do not like how it sounds, but yes. There was rest in it.

SOCRATES: Do not be ashamed. Every man wants that rest, and that is why every man looks for a mentor in such a hurry. It is not only knowledge we seek; it is someone on whom to lay the weight of judging. And notice: as long as you had a single master, the rest worked. What broke it was the second one.

GLAUCON: Because now one says one thing and the other says another.

SOCRATES: No. On most things, the two say the same, or you would not admire both. What broke your rest was something else: you discovered that the admiration you feel for one does not oblige the other to feel the same. You wanted your masters to love each other as you love them. They have no such duty.

GLAUCON: But if Damon is wise, and says Theophilus is a charlatan, should I not at least consider it?

SOCRATES: You should, and you should also consider the opposite. But let us go slowly. When Damon teaches you geometry, where does his authority come from?

GLAUCON: From knowing geometry.

SOCRATES: And when he tells you Theophilus is a charlatan, where does the authority come from?

GLAUCON: From… knowing Theophilus, I suppose.

SOCRATES: And does he? Has he heard him teach? Has he read what he writes? Or does he know the reputation, a sentence spoken in an argument, the memory of an old feud nobody remembers how it started?

GLAUCON: I do not know. I never asked.

SOCRATES: Then here is the first thing to keep. A man can be a master in one art and as ignorant as anyone in a second. Knowing geometry makes no one a judge of character. When you accept Damon’s judgment of Theophilus with the same trust you give the theorem, you are extending his authority into ground where he never earned it. And he, if honest, would not claim it.

GLAUCON: So neither one is wrong about geometry. They are wrong about each other.

SOCRATES: Perhaps. Or perhaps one of them is right, and you do not yet know. But notice what you just did: a minute ago, the problem was “which of the two to follow”. Now it is “what does each one actually know, and about what”. It is a better question.

GLAUCON: It is a question that hands the work back to me.

SOCRATES: That work was always yours, Glaucon. You had only lent it out. We talked, on another afternoon, about the tablet in the square: the label someone sticks on you does not excuse you from examining. The same holds for the label one master sticks on another. “Charlatan” and “old man who repeats” are thrown stones, not reasons. Ask for the reasons. If they come, weigh them. If they do not, leave the stones on the ground.

GLAUCON: And if I weigh them and conclude that Damon, who taught me so much, was wrong about Theophilus? What do I do with that?

SOCRATES: What do you do when your horse stumbles?

GLAUCON: I get it up and carry on. I do not sell the horse.

SOCRATES: And why would you sell the master? Glaucon, listen well to this, because it is the part that costs the young the most: your master will err. Not “may err”, will. He will err about people, he will err about facts, he will have days of jealousy, of tiredness and of stubbornness. And when that happens, you will have two wrong ways out and one right one. The first wrong one is pretending you did not see, to protect the rest. The second wrong one is tearing down the whole man, because one column cracked. The right one is harder: keep learning geometry from the one who knows geometry, and keep for yourself the judgment about people.

GLAUCON: It sounds like having less trust in him.

SOCRATES: It sounds like having the right trust, instead of whole trust. Whole trust is what one gives the gods, and not even them that much. Giving a man the trust that belongs to a god is not honoring him. It is condemning him to disappoint you, because he has no way to pay a debt you yourself invented.

GLAUCON: So the mentor is a man like me.

SOCRATES: Who went a little further down a road. Only that, and that is already a lot. Notice that it is precisely the error that lets you recognize an equal in him. A master who never erred would not be a master, he would be an oracle, and before oracles one does not learn, one obeys. A man who errs and corrects, who shows where he stumbled, teaches you twice: he teaches the art and he teaches that the art fits inside a human life. The error is the proof that the road he walked is a road you can also walk.

GLAUCON: I had never thought of the master’s error as a gift.

SOCRATES: It is not a gift, it is a fact. But it is a fact that frees you, if you accept it. As long as your master is perfect, you will be a pupil forever. On the day you see him err, and still respect him anyway, you will have begun to have an opinion of your own. And having your own opinion about people, including those you most admire, is the beginning of being someone, and not someone’s shadow.

GLAUCON: And Theophilus? Do I keep listening to him?

SOCRATES: Keep listening to both. Learn from each what each one knows. And when one speaks ill of the other, do what you would do with any other claim: ask “why?”, listen, and decide for yourself. It is possible that one day you will find one of them was right. It is possible you will find both were being small, as men sometimes are. In either case you will have learned something neither of them could teach you: that your eyes count too.

GLAUCON: And if my eyes err?

SOCRATES: They will, and then you will be like them. That is what makes us human, Glaucon: not the absence of error, but the duty to judge even knowing we may err, and the decency to keep honoring those who erred before us.

..then Glaucon put both scrolls in the same bag, Socrates remarked that the bag looked lighter, and the two went for a beer.

Glossary

• agora: the central square of Greek cities, where the people gathered
• papyrus: the material people wrote on before paper, kept in rolled scrolls
• oracle: a place or person that, in ancient Greece, gave answers held to come from the gods

Note

This dialogue is a continuation of the Dialogue on Labels and Truth. There, the question was the label others stick on us. Here, it is the label the people we admire stick on one another.

Everyone looks for a mentor, and the need is legitimate: nobody learns everything alone. The problem is not in trusting, it is in the amount. It is very easy to hand someone we admire not only what they know, but also what they think, including about other people. And when two people we respect do not respect each other, the comfort of having a guide collapses, because we discover that their judgment of the world was never a single one.

The way out is neither picking a side nor abandoning both. It is accepting that everyone errs, mentor included, and that having our own opinion about people, formed with our own reasons and our own eyes, is not disrespect. It is the only form of respect that does not demand the other be more than human.

I have always admired medicine, especially those doctors who devote their efforts, their time and every cell of their own bodies to the pursuit of excellence. Among the specialties, one has always stirred my admiration: cardiology.

One of the names I carry as a reference is Enéas Carneiro. Many admired him for his intelligence, but I always saw in him something I consider even more important and less debatable: will.

Enéas had will. He knew what he wanted, defended his convictions and wished for a better Brazil. Beyond that, he dedicated part of his life to the study of the heart, becoming a cardiologist.

But life found a way to turn that admiration into something much deeper.

On November 13, 2025, my daughter Luna was born with a severe congenital heart disease. She had to undergo heart surgery at Hospital Vila da Serra, in Nova Lima, Minas Gerais.

At that moment, everything I admired in medicine stopped being something distant, almost abstract, and became part of my own story.

What I once admired, today I venerate.

I believe God blessed the team of cardiologists at that hospital. Men and women who devoted their lives to knowledge, to science and to the refinement of their skills and who, through everything the human being has been able to learn and develop, were given the opportunity to save my daughter’s life.

It is hard to find words that express the size of my gratitude. There is something deeply admirable in someone who dedicates their own existence to acquiring knowledge and skills that, one day, may mean the difference between life and death for another person.

I leave here my deepest respect and my eternal gratitude to healthcare professionals, especially the team that cared for Luna, and to everyone inside hospitals driven by a true intention and a greater purpose: saving lives.

If before I admired medicine for the knowledge and excellence of its professionals, today I also admire it for something no theory could ever teach me: the chance to keep watching my daughter grow.

“The most important fact that sets the human being apart from all its brothers and cousins on the phylogenetic scale is knowledge; only knowledge frees man, only through knowledge is man free and, being free, he can aspire to a better condition of life for himself and all his fellow beings. I can only understand a society in which knowledge is the foremost reason the government gives for the formation of the citizen. My message is a positive one: that man must know, must understand, and in understanding, he is free.”

— Dr. Enéas Ferreira Carneiro (Rio Branco, November 5, 1938 — Rio de Janeiro, May 6, 2007)

How to use it: not every repo needs every item. Items marked (A) are advanced, worth it for OSS projects, monorepos, or anything with external contributors. Everything unmarked is the baseline.

Sections marked conditional only apply when the repo is that kind of project (an HTTP service, a containerized app, a Kubernetes workload).

Index

• 1. Documentation
  • 1.1 README.md
  • 1.2 Governance and policy files
  • 1.3 Deeper docs (docs/)
  • 1.4 Changelog and releases
• 2. Code
  • 2.1 Layout and conventions
  • 2.2 Configuration
  • 2.3 Single source of truth / generated code (A)
• 3. Testing
  • 3.1 Unit tests
  • 3.2 Integration and end-to-end
  • 3.3 Performance and robustness (A)
  • 3.4 Coverage
• 4. API artifacts (conditional: HTTP/gRPC services)
• 5. Build and runtime (conditional: containerized apps)
  • 5.1 Docker
  • 5.2 Docker Compose
  • 5.3 Kubernetes (conditional)
• 6. CI/CD
  • 6.1 Basic checks
  • 6.2 Security
  • 6.3 Dependencies
  • 6.4 Repository automation (A)
• 7. Repository metadata
• Minimum viable set

1. Documentation

1.1 README.md

• Project overview: what it does and, more importantly, why it exists
• Status badges: CI build, test coverage, latest release, stability level
• Stability / maturity level (development → alpha → beta → stable → deprecated)
• Support policy: who maintains it, where to ask, expected response time
• Installation instructions
• Running the app locally
• Configuring your environment (.env.example committed, .env git-ignored)
• Usage examples (copy-pasteable, actually tested)
• Architecture diagram or a short “how it fits together” section
• Link to contribution guidelines
• License information
• A README per module/subpackage, not only at the root (A)
• Autogenerated status sections, delimited by markers, so they never drift (A)

1.2 Governance and policy files

• LICENSE (+ NOTICE if you redistribute third-party code)
• CONTRIBUTING.md: local setup, how to run tests, PR title/description rules, review expectations
• CODEOWNERS: ownership per directory, drives automatic review requests
• SECURITY.md: how to report a vulnerability, privately
• CODE_OF_CONDUCT.md
• AGENTS.md / CLAUDE.md: AI-assisted contribution policy (disclosure trailer, what agents may and may not do)
• .github/pull_request_template.md
• .github/ISSUE_TEMPLATE/: bug report, feature request, plus config.yml pointing to support channels
• Issue triage policy: labels, priorities, stale policy (A)
• Maintainer/approver roles and how to become one (A)

1.3 Deeper docs (docs/)

• Release process, written down and repeatable
• Testing guide: how to run each test tier locally
• Onboarding guide for new modules/components (A)
• ADRs (Architecture Decision Records): the why behind the design
• Runbook / operational docs: alerts, dashboards, common failures
• Generated reference docs: every config option, default, and emitted metric, produced from the source of truth rather than hand-written (A)

1.4 Changelog and releases

• CHANGELOG.md following SemVer / Keep a Changelog
• Changelog fragments: one small file per PR, assembled at release time. Kills merge conflicts and forces the author to state user impact (A)
• Separate changelog for API/breaking changes vs. user-facing changes (A)
• Tagged releases with release notes generated from the changelog
• Documented versioning and deprecation policy

2. Code

2.1 Layout and conventions

• Consistent, documented project layout
• License/SPDX header on every source file, enforced by the linter
• Package-level doc comment on each package (doc.go or equivalent)
• Linter config committed, with an explicit linter set and justified per-path exceptions, not the tool’s defaults
• Formatter enforced in CI, not left to the reviewer
• Import restrictions enforced (no forbidden/internal imports leaking out) (A)
• A task runner (Makefile / justfile / npm scripts) with standard targets: test, lint, fmt, build, run, integration-test, tidy
• The same commands work at the root and inside each module (A)

2.2 Configuration

• Typed config struct with an explicit Validate() and clear error messages
• Fail fast on invalid config at startup, never mid-request
• Committed example config, loaded by a real test
• Published config schema (JSON Schema / equivalent) for editor autocomplete and CI validation (A)
• examples/ with complete, scenario-based configurations: not snippets
• Secrets never in config files or in the repo; injected at runtime
• Structured logging, metrics, and tracing wired in and documented

2.3 Single source of truth / generated code (A)

• Declarative metadata file per component (owners, stability, config, emitted telemetry)
• Code, docs, and README sections generated from it
• CI verifies the generated output is up to date: drift becomes a failing build instead of stale documentation

3. Testing

3.1 Unit tests

• Unit tests that actually assert behavior, not just execute lines
• Table-driven tests for branchy logic
• Deterministic: no sleeps, no real clock, no network
• Test fixtures (testdata/) versioned, with a command to regenerate them

3.2 Integration and end-to-end

• Integration tests isolated behind a build tag / separate command so they don’t slow the unit run
• Real dependencies via ephemeral containers (Testcontainers or equivalent)
• End-to-end tests against a realistic deployment (A)
• Contract tests for consumed and provided APIs (A)

3.3 Performance and robustness (A)

• Benchmarks for hot paths, with a documented way to compare runs
• Load tests with a stated performance baseline
• Fuzz tests for anything that parses external input
• Leak detection: goroutines, connections, file descriptors
• Race detector enabled in CI

3.4 Coverage

• Coverage report generated in CI and published
• Coverage threshold committed to the repo, split into project and patch targets (e.g. 85% overall, 95% on the diff)
• Per-module/per-component coverage, so a large repo doesn’t hide a bad corner behind a good average (A)

4. API artifacts (conditional: HTTP/gRPC services)

• OpenAPI / Swagger spec, versioned in the repo
• Spec generated from or validated against the code, so it can’t drift
• Postman / Insomnia / Bruno collection
• .http request files (runnable from the editor)
• curl.txt with copy-pasteable examples
• Documented authentication flow
• Error catalogue: codes, meanings, and how to recover
• Documented versioning and deprecation path for endpoints

5. Build and runtime (conditional: containerized apps)

5.1 Docker

• Dockerfile with multi-stage build
• Minimal, pinned base image (distroless/slim), non-root user
• .dockerignore
• Healthcheck defined
• Reproducible builds: pinned dependencies, no latest
• Image vulnerability scan in CI (A)
• SBOM generated and published with the image (A)

5.2 Docker Compose

• docker-compose.yml that brings the whole stack up with one command
• Environment-specific overrides (docker-compose.dev.yml, .prod.yml)
• Dependencies (DB, broker, cache) included so a new dev needs nothing else

5.3 Kubernetes (conditional)

• Manifests, or a Helm chart with a documented values.yaml
• Resource requests and limits set
• Liveness / readiness / startup probes
• Secrets management (External Secrets, SOPS, sealed-secrets, never plain)
• Ingress configuration
• Per-environment overlays (Kustomize) or value files
• NetworkPolicy, PodDisruptionBudget, security context (A)
• Manifests linted and validated in CI (kubeconform, helm lint) (A)

6. CI/CD

6.1 Basic checks

• Lint
• Format check
• Unit tests
• Coverage report + threshold enforcement
• Build the artifact (binary and/or image)
• Integration tests, in a separate job
• Required status checks configured on the protected branch
• Build matrix across supported OS/versions (A)
• Scope detection: only run what the diff actually affects (A)
• Job sharding to keep wall-clock time sane on large repos (A)

6.2 Security

• Dependency vulnerability scanning
• SAST (CodeQL or equivalent)
• Secret scanning enabled
• License compliance check on dependencies
• Workflow files themselves linted (actionlint, zizmor) (A)
• Shell scripts linted (shellcheck) (A)
• Minimal permissions: per workflow; third-party actions pinned to a SHA
• OpenSSF Scorecard (A)

6.3 Dependencies

• Automated updates (Renovate / Dependabot) with a grouping strategy
• Lockfile committed
• Automated tidy/prune check so the manifest can’t drift
• Public API compatibility check between versions (A)

6.4 Repository automation (A)

• Auto-assign reviewers from CODEOWNERS
• Auto-label PRs and issues by affected area
• Ping owners on new issues touching their area
• Stale issue/PR handling
• First-time contributor welcome
• Merge freeze during a release window
• Periodic repo health report (open issues, unowned areas)
• Release preparation automated (version bump, changelog assembly, tag)

7. Repository metadata

• Topics/tags with the technologies used, for discoverability
• Clear, one-line repository description
• Branch protection rules on the default branch
• Signed commits or signed tags (A)
• Archive/deprecation notice when a repo stops being maintained

Minimum viable set

If you only do ten things, do these:

1. README that gets someone running locally in under five minutes
2. LICENSE
3. CONTRIBUTING.md
4. .env.example committed, secrets out of the repo
5. Unit tests with a coverage threshold enforced in CI
6. Lint + format enforced in CI
7. Multi-stage Dockerfile and a one-command docker-compose up
8. CHANGELOG.md and tagged releases
9. Dependency and vulnerability scanning
10. Branch protection with required checks

GLAUCON: Socrates, I found the tool that solves everything. I read the documentation last night, got an example running in twenty minutes and it worked on the first try. I am going to rewrite the whole system on top of it.

SOCRATES: Twenty minutes. When you test drive a car, how long does it last?

GLAUCON: About fifteen minutes. A lap around the block with the salesman beside you.

SOCRATES: And have you ever found a defect in a car on a lap around the block?

GLAUCON: Never. Defects show up later. On the highway, in the rain, in stalled traffic.

SOCRATES: So in those twenty minutes, did you get to know the tool, or did you get to know the route its author chose for you to take?

GLAUCON: The route. But I liked the route.

SOCRATES: Liking it is already something. Let me ask you a few questions, then. Just do not answer with what you imagine. Answer with what you know.

GLAUCON: Go ahead.

SOCRATES: How many records went through it in your test?

GLAUCON: About a thousand.

SOCRATES: And on an ordinary day, in production?

GLAUCON: About ten million.

SOCRATES: And what does it do when it cannot keep up? Hold what is coming in, drop it, or go down with it?

GLAUCON: I do not know. I never got to that point.

SOCRATES: Nobody does, in a test. And notice: scale does not invent any defects. It promotes the ones that were already there and nobody saw. A slow operation over a thousand records is invisible. Over ten million, it is the whole system. You will not discover new defects in production, you will discover the real size of the ones you already had.

SOCRATES: What language was it written in?

GLAUCON: Does it matter? It runs in a container, I do not even need to know.

SOCRATES: It is the one part of it you will not be able to swap later. The language has already decided, before you arrived, a lot of things you will only feel in production. Which is it?

GLAUCON: Rust.

SOCRATES: Then it is compiled, and that gives you good things for free. It will arrive as a single file, with no interpreter or loose dependency to carry along, and it starts in a blink. An interpreted tool forces you to look after it and its interpreter for the rest of your life.

GLAUCON: Great. So it is the right choice.

SOCRATES: It is a choice, and every choice has the other side facing down. That single file carries inside it the libraries the author chose, in the versions he chose, on the day he compiled it. When a problem shows up in one of those libraries, you do not swap the part. You wait for him to compile again.

GLAUCON: I had not thought of that.

SOCRATES: It is the same package, looked at from the other side. Second thing: Rust has no garbage collector.

GLAUCON: That is good, isn’t it?

SOCRATES: It is good for one specific thing. A language with a collector has a janitor who cleans the house when he decides to, not when you ask, and that shows up as a slowness that arrives unannounced and leaves on its own, right when volume grows. Without a collector, response time is predictable, and predictable is worth a lot.

GLAUCON: So I have nothing to worry about regarding memory.

SOCRATES: You do, and of a quieter kind. Without a collector nothing gets swept up by accident: whatever the tool holds, it holds until the end. An unbounded queue or a cache nobody told to stop do not show up as a pause, they show up as a line slowly climbing for a week until someone gets woken up in the middle of the night. A garbage collector is noisy and warns you. Its absence is silent.

GLAUCON: Can that be known beforehand?

SOCRATES: Two questions settle it. How much memory it consumes after a week running, and not in the first ten minutes. And how many cores it actually uses, because it is one thing for the language to know how to use eight, and another for the tool to have been written for that.

GLAUCON: That changes the whole design.

SOCRATES: It does, and it is far better to change it now, while it is still paper. Next: how does a person prove to it that they are who they say they are?

GLAUCON: Username and password, created right there inside it.

SOCRATES: And in your company, how do people prove that everywhere else?

GLAUCON: Through LDAP.

SOCRATES: Then it will be the only place in the company where a person exists twice, and whoever gets let go on a Friday will leave every place except one. Now the other half: after the person gets in, who decides what they can do?

GLAUCON: Whoever gets in can do everything, I think.

SOCRATES: You just treated as one thing what are two. One is who gets in. The other is what each one does after getting in. A tool for a single person does not need to separate the two, because the person is you.

GLAUCON: It is just that nobody on my team would cause damage on purpose.

SOCRATES: It is almost never on purpose. Permissions do not exist out of distrust, they exist to limit the size of a distracted person’s mistake on a Tuesday. When everyone can do everything, everyone carries the power to bring everything down, and nobody asked for that power.

GLAUCON: I had never thought of it that way.

SOCRATES: And when someone deletes what they should not have, do you find out who it was?

GLAUCON: I do not know whether it keeps that.

SOCRATES: Ask. A log does not exist to accuse anyone, it exists precisely so that nobody needs to be accused. Without it, on the day of the problem everyone who had access is a suspect, and the conversation becomes memory against memory. Memory is the most biased witness there is.

SOCRATES: When did its last version come out?

GLAUCON: About two months ago, I think.

SOCRATES: And the one before that?

GLAUCON: I did not look.

SOCRATES: Look, because the interval between versions is the clock you will tie yours to. And look at something finer: between one big version and the next, do small versions come out, with fixes only?

GLAUCON: I did not notice.

SOCRATES: Notice. If they do, on the day of a serious problem you update a decimal place and go back to sleep. If they do not, the fix only arrives together with the next big version, with everything else it changes thrown in, and an afternoon’s problem becomes a whole migration on a day you did not choose.

GLAUCON: That I can check today.

SOCRATES: Check it along with this one: how many people actually maintain it?

GLAUCON: The repository has eight thousand stars and hundreds of contributors.

SOCRATES: A star is applause, and applause fixes nothing. A contributor, on most of those lists, is anyone who ever fixed a comma in the documentation. Count something else: in the last six months, how many different names touched the core of the tool? And of those, how many are left after you remove the first one?

GLAUCON: I do not know. I imagine few.

SOCRATES: If one is left, you did not choose a tool. You chose a person, and that person signed nothing with you.

GLAUCON: Are there many questions left?

SOCRATES: One. You said it does almost everything you need. And the almost?

GLAUCON: Whatever is missing, I open an issue and ask.

SOCRATES: Always? Every time a new need appears, the path will be to ask and wait?

GLAUCON: It is what can be done.

SOCRATES: Then notice what you accepted: the pace of what you deliver becomes decided by people who do not know your problem and owe you nothing. There is another path, and it is the difference between a tool and a wall: a place where you yourself write what was missing. A plugin, a hook, a script it calls at the right moment. When that place exists, what is missing today is your work. When it does not, what is missing today is waiting.

GLAUCON: Socrates, I could hardly answer anything.

SOCRATES: And it is the best thing that happened to you today. Notice that you did not discover that the tool is bad. You discovered that you do not know it. Those are very different things, and only one of them can be fixed.

GLAUCON: But I was confident.

SOCRATES: You were confident of what you saw, and you saw twenty minutes chosen by someone else. Everyone is confident at the start. The difference is that some discover the questions before installing, and others discover them afterwards, with the whole system standing on top of them.

GLAUCON: So I give up on it?

SOCRATES: That is not what I said. Maybe it is excellent. Go after every answer and come back with them. If they are good, you install it knowing what you installed. If they are bad, you already know where it will hurt, and sometimes you install it anyway, on purpose, because hurting in that spot is acceptable.

GLAUCON: And if I am wrong even after asking everything?

SOCRATES: You will be wrong now and then, and that is fine. Put the new thing where a mistake is cheap, on an edge you can switch off in an afternoon without waking anyone, and keep the middle boring and predictable. Before asking what a tool does, ask how much it costs to leave it. That is the one answer nobody writes in the documentation.

GLAUCON: I am going to redo the diagram.

SOCRATES: Not today. Keep it for a week and open it again. Enthusiasm is good fuel and a terrible driver.

..then Socrates closed the laptop, Glaucon saved the diagram without deleting anything, and the two went for a beer.

Note

The idea was to take the dialogue form of Plato’s Republic and apply it to a technology case. Only the form is borrowed: short questions, one idea at a time, the interlocutor’s answer opening the next question.

The case is familiar to anyone who works with this. Every week a new tool appears, open source, with a pretty interface, an example that boots in five minutes and a star chart going up. At that point everything seems solved, and the enthusiasm is legitimate.

Things get more complicated when the questions begin. How it behaves at scale. What happens on the day you need something the author did not foresee. How long a bug report takes to become a fix, and whether anyone even replies. How many real maintainers stand behind the repository numbers, and from how many different companies. Who decides the direction of the project, and where that decision is discussed. How it does authentication, and whether that talks to the login the rest of the company already uses. What it does when a piece of it goes down. Whether you can audit afterwards who did what inside it. And how much work it actually is to integrate it with what you already have running.

Almost none of these answers are on the first page. Most show up about six months later, when switching is no longer free.

Reference architectures

Three ways to wire the tools above into a working stack. All of them have the same three layers, ingestion, processing and storage, plus whatever consumes the result.

1. Batch lakehouse

Nightly and hourly ELT. The default when latency is measured in hours.

  flowchart TB
  subgraph ing["Ingestion"]
    airbyte["Airbyte"]
    debezium["Debezium + Kafka Connect"]
  end
  subgraph proc["Processing"]
    spark["Apache Spark"]
    dbt["dbt"]
    ge["Great Expectations"]
  end
  subgraph sto["Storage"]
    iceberg["Apache Iceberg"]
    s3["MinIO / S3"]
    polaris["Apache Polaris"]
  end
  subgraph out["Consumption"]
    trino["Trino"]
    superset["Apache Superset"]
  end

  airbyte --> spark
  debezium --> spark
  spark --> dbt --> ge --> iceberg
  s3 -.- iceberg
  polaris -.- iceberg
  iceberg --> trino --> superset

Apache Airflow sits outside the picture, triggering every arrow above.

2. Streaming

Seconds instead of hours. Two branches out of one job: the table for history, the OLAP store for the dashboard.

  flowchart TB
  subgraph ing["Ingestion"]
    debezium["Debezium"]
    kafka["Apache Kafka"]
  end
  subgraph proc["Processing"]
    flink["Apache Flink"]
  end
  subgraph sto["Storage"]
    paimon["Apache Paimon on S3"]
    clickhouse["ClickHouse"]
  end
  subgraph out["Consumption"]
    trino["Trino"]
    grafana["Grafana"]
  end

  debezium --> kafka --> flink
  flink --> paimon --> trino
  flink --> clickhouse --> grafana

Airflow is still needed here, not to schedule the job but to run the compaction and snapshot expiration the table needs because the job never stops writing.

3. ML and inference

Features written by the same pipeline that feeds training, and a model served out of a registry.

  flowchart TB
  subgraph ing["Ingestion"]
    kafka["Apache Kafka"]
  end
  subgraph proc["Processing"]
    flink["Apache Flink"]
    spark["Apache Spark"]
  end
  subgraph sto["Storage"]
    feast["Feast"]
    delta["Delta Lake on S3"]
    qdrant["Qdrant"]
  end
  subgraph train["Training"]
    ray["Ray"]
    mlflow["MLflow Registry"]
  end
  subgraph serve["Serving"]
    kserve["KServe"]
    vllm["vLLM"]
  end

  kafka --> flink
  kafka --> spark
  flink --> feast
  flink --> qdrant
  spark --> delta
  delta --> ray --> mlflow --> kserve
  feast --> kserve
  qdrant --> vllm

Official repositories

• Airbyte: Connector-based ELT for loading data into a warehouse or lake.
• Amundsen: Data discovery and metadata search for analysts.
• Apache Airflow: Schedules and monitors pipelines written as Python DAGs.
• Apache Arrow: In-memory columnar format shared between engines and languages.
• Apache Atlas: Metadata and lineage governance for the Hadoop ecosystem.
• Apache Avro: Row-oriented serialization format that carries its schema.
• Apache Beam: One pipeline model that runs on several batch and streaming engines.
• Apache Doris: MPP analytical database aimed at real-time reporting.
• Apache Flink: Stateful stream processing with event time and exactly-once.
• Apache Hudi: Lake table format with upserts and incremental reads.
• Apache Iceberg: Open table format with ACID commits, schema evolution and time travel.
• Apache Kafka: Distributed log for publishing and consuming event streams.
• Apache NiFi: Flow-based routing and transformation with a visual editor.
• Apache ORC: Columnar file format built for large Hive-style reads.
• Apache Paimon: Lake table format designed to be written continuously by Flink.
• Apache Parquet: The columnar file format most of the lake is stored in.
• Apache Polaris: Open catalog so several engines see the same Iceberg tables.
• Apache Pulsar: Messaging and streaming with storage separated from serving.
• Apache Spark: General engine for batch, SQL and ML over large datasets.
• Apache Superset: Web BI with dashboards and SQL exploration.
• Argo Workflows: Runs container-native workflows as Kubernetes objects.
• BentoML: Packages a model and its dependencies into a service.
• ClickHouse: Column store for fast analytical queries at scale.
• Dagster: Orchestrator built around data assets rather than tasks.
• Dataform: SQL transformation workflows with dependency management.
• DataHub: Metadata platform for catalog, lineage and ownership.
• dbt: Turns SQL selects into tested, documented and versioned models.
• Debezium: Reads database change logs and publishes them as events.
• Deequ: Data quality checks expressed as Spark jobs.
• Delta Lake: Transactional storage layer over Parquet files.
• Dremio: Query engine and semantic layer over lake storage.
• DuckDB: Analytical database that runs inside your process, no server.
• Feast: Feature store serving the same features to training and inference.
• Flyte: Typed, versioned workflows for data and ML on Kubernetes.
• Grafana: Dashboards over metrics, logs and traces.
• Great Expectations: Declares expectations about data and validates them.
• Hive Metastore: The table and partition catalog many engines still read from.
• Hopsworks: Feature store and ML platform.
• Jupyter: Notebooks for interactive analysis.
• KServe: Serves models on Kubernetes with autoscaling and canaries.
• Kubeflow: ML platform assembled from Kubernetes components.
• Kubeflow Pipelines: Composes and runs ML pipelines as containers.
• LightGBM: Gradient boosting tuned for speed on large tabular data.
• Marquez: Reference server that collects and stores OpenLineage events.
• Metabase: BI that lets non-engineers ask questions of the database.
• Metaflow: Python workflows for data science, from Netflix.
• Milvus: Vector database for similarity search at scale.
• MinIO: S3-compatible object storage you run yourself.
• MLflow: Tracks experiments, packages models and holds the registry.
• NumPy: Arrays and numerical computing, the base of the Python stack.
• NVIDIA Triton: Inference server for models on GPU and CPU.
• OpenLineage: Open standard for pipelines to report what they read and wrote.
• OpenMetadata: Catalog, lineage, quality and governance in one place.
• OpenTelemetry: Vendor-neutral standard for traces, metrics and logs.
• Pandas: DataFrames for data manipulation in Python.
• pgvector: Vector types and indexes inside PostgreSQL.
• Polars: Fast DataFrames written in Rust, with a lazy query engine.
• Prefect: Orchestration for Python workflows, built for dynamic runs.
• Presto: Distributed SQL engine that queries data where it lives.
• Project Nessie: Git-like branches and commits for lake tables.
• Prometheus: Time series database and alerting for metrics.
• Python: The language most of this stack is written and driven in.
• PyTorch: Deep learning framework with dynamic graphs.
• Qdrant: Vector search engine with filtering, written in Rust.
• R: Language and environment for statistics.
• Ray: Distributed Python runtime for training, tuning and serving.
• Redpanda: Kafka-compatible broker with no JVM and no ZooKeeper.
• Scikit-learn: Classical machine learning algorithms in Python.
• SciPy: Scientific computing built on NumPy.
• Seldon: Deploys and monitors models on Kubernetes.
• Soda: Data quality checks written as readable rules.
• SQLMesh: Transformations with column-level lineage and virtual environments.
• StarRocks: MPP OLAP database for sub-second queries over the lake.
• TensorFlow: Deep learning framework with a production toolchain.
• Trino: Distributed SQL that federates queries across many sources.
• vLLM: High-throughput serving for large language models.
• Weaviate: Vector database with built-in embedding modules.
• XGBoost: Gradient boosting library, the workhorse for tabular data.

Kafka Connect ships inside apache/kafka, dbt Tests inside dbt-core and Ray Serve inside ray-project/ray, so they share the repositories above.

No public repository, these are proprietary or managed services: Power BI, Tableau, Fivetran, Amazon S3, Google Cloud Storage, Azure Data Lake Storage, AWS Glue Catalog, Databricks, Starburst, Snowflake, BigQuery, Redshift, Tecton, Pinecone, Weights & Biases, Neptune, Comet, Vertex AI, SageMaker and Azure ML.

• Index
• Introduction
• My goal
• The problem Flink solves
• Real use cases
• The architecture, in a few pieces
• Sources and sinks
• Parallelism
• Redistribution and data skew
• Operator chaining
• Rich functions
• Time: the part that breaks intuition
• Watermarks
• Windows
• ProcessFunction
• State
• Checkpoints and savepoints
• Flink SQL
• In praxi (“in practice”)
• Deployment
  • Execution modes
  • Kubernetes
  • Memory
  • CPU and slots
  • Local disk
  • S3 and the remote file system
• The ecosystem around it
• Complexities
• What stayed with me

Introduction

This text is the result of what I studied and experimented with over this week. It is not a tutorial by someone who has mastered Apache Flink, nor does it intend to be a reference on the subject. It is more like the notebook of someone who is learning, testing a few things in practice and sharing what they managed to understand and discover so far.

I have always believed that explaining is the best way to find out what we really understood. When I write, the holes show up on their own..

Apache Flink is interesting mainly because it treated streaming as the general case, and not as a patch on top of batch processing. While its competitors at the time simulated continuous flow with micro-batches, Flink processes record by record, with low latency and high throughput, keeping distributed state consistent through asynchronous snapshots (a variation of the Chandy-Lamport algorithm) that guarantee exactly-once semantics even after failures. It also offers a notion of event time with watermarks, which allows it to correctly handle data that arrives out of order or late, something essential in the real world. Its origin is academic: it was born in 2010 as Stratosphere, a research project at TU Berlin in partnership with Humboldt University and the Hasso Plattner Institute, led by Volker Markl. In 2014 the code was donated to the Apache Software Foundation, renamed Flink (“nimble” in German, hence the squirrel in the logo) and promoted to a top-level project in December of that same year. The creators founded data Artisans to commercialize it, a company bought by Alibaba in 2019 and renamed Ververica. Alibaba itself already maintained an internal fork (Blink) to handle the volume of Singles’ Day, and a good part of those improvements went back into the main project, helping consolidate Flink as the de facto standard for stream processing at companies like Netflix, Uber and Stripe.

My goal

My goal is to use Apache Flink connected to Kafka to process OpenTelemetry OTLP data (traces, metrics and logs) and extract more intelligence from that telemetry in real time, instead of just storing it and querying it later. The idea is to use this pipeline as a lab to test cases such as anomaly detection in time series of metrics and latencies, alert generation with context and noise suppression, root cause analysis (RCA) correlating spans, logs and metrics of affected services, prediction of degradation and resource saturation before the incident happens, and detection of sensitive data improperly flowing through span attributes and log messages. Beyond that, I want to explore other scenarios where Flink fits well: enrichment and smart sampling of traces, deciding with state which traces to keep based on what happened in the entire request (tail-based sampling) instead of discarding randomly at the edge; continuous calculation of SLIs and error budgets per service, with sliding windows and aggregations by endpoint, team and deploy version; and detection of behavior changes after releases, comparing the traffic, error and latency profile between versions to identify regressions introduced by a specific deploy. I intend to detail all of this soon (:

The problem Flink solves

For a long time the data world was organized around the batch. You accumulate data during the day, run a job at midnight, and in the morning the report is ready. That works, and keeps working for a lot of things. The problem is when the answer needs to come earlier. Detecting a card fraud ten hours after the purchase is the same as not detecting it. An infrastructure alert that arrives the next day is a post-mortem, not an alert. In those cases, you do not want to ask the data from time to time, you want the data to answer continuously. That is the inversion that stream processing proposes:

• Batch: the query is permanent, the data is finite. You run the query against a set that is standing still.
• Streaming: the data is infinite, the query is what stands still. It is registered once and reacts to every event that passes.

Apache Flink is a distributed processing engine aimed at building that second kind of application. Its founding premise: everything is a stream. Yesterday’s CSV file is a stream that has a beginning and an end (bounded). The Kafka topic with clicks from your site is a stream that never ends (unbounded). One of the points that caught my attention most during the study was the unified processing model: in Flink, batch can be understood as a particular case of streaming, in which the data flow has a defined beginning and end. This means the execution model does not depend on two completely distinct engines, but on a single architecture capable of processing both continuous flows and finite datasets.

Real use cases

It is worth looking at where this shows up in practice, because the list helps understand what kind of problem Flink solves well:

• Fraud detection, in banks and fintechs. Every transaction is evaluated in milliseconds against the customer’s history, kept as state per key (keyBy on the customer id). Rules like “3 purchases in different cities within 10 minutes” are expressed with windows, timers and the complex event pattern detection library (CEP).
• CDC pipelines to a data lake. Flink CDC follows the database transaction log (the binlog in MySQL, the WAL via logical replication in Postgres) and replicates every insert, update and delete in real time to table formats like Apache Iceberg, Paimon or Hudi. Stripe is one of the public cases of change capture at scale with Flink.
• Real-time analytics and dashboards. Windowed aggregations (orders per minute, revenue per region) feeding live dashboards. Uber, Netflix and Alibaba operate thousands of Flink jobs with this profile.
• Event-driven applications. Here the business logic lives inside the stream, and not in a service that queries a database: ride matching, dynamic pricing, alerting engines, feature store updates for machine learning models.
• AI over streams. Model inference applied to events as they pass, and Flink Agents, a new subproject aimed at AI agents that react to events in real time.

The architecture, in a few pieces

A job is represented as an operator graph, composed of sources (source), transformations (map, filter, keyBy, window) and sinks (sink). From that graph, Flink determines the parallelism of each operator and distributes its instances among the slots available in the cluster.

keyBy deserves special attention because it is responsible for partitioning the data. When you define keyBy(event -> event.getUserId()), you are determining that events with the same key are routed to the same parallel instance of the next operator. This partitioning is fundamental for keyed stateful processing, because it allows each instance to keep and manage the state associated with the keys under its responsibility, without the need for distributed coordination on every event.

Sources and sinks

Source and sink are the two points where the job touches the world. In the 2.x line they go through a single API, env.fromSource(...) on one side and stream.sinkTo(...) on the other.

The three sources that appear in almost every project:

And the three sinks:

The shape of the code is always the same, a builder on one side and another on the other:

KafkaSource<String> source = KafkaSource.<String>builder()
    .setBootstrapServers("kafka:9092")
    .setTopics("otlp-spans")
    .setGroupId("flink-otlp")
    .setStartingOffsets(OffsetsInitializer.earliest())
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();

KafkaSink<String> sink = KafkaSink.<String>builder()
    .setBootstrapServers("kafka:9092")
    .setRecordSerializer(KafkaRecordSerializationSchema.builder()
        .setTopic("otlp-alerts")
        .setValueSerializationSchema(new SimpleStringSchema())
        .build())
    .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .setTransactionalIdPrefix("otlp-")
    .build();

env.fromSource(source, watermarkStrategy, "otlp-spans")
   .process(new AnomalyDetector())
   .sinkTo(sink);

Two things in this code are not syntax details. The first is that the source does not commit offsets on its own: the offset is Flink state, it goes into the checkpoint together with everything else and comes back with it on recovery. That is why you do not manage offsets by hand here, and it is also why Kafka’s consumer lag dashboard may disagree with the job’s reality, since the commit to Kafka is only informational.

The second is that exactly-once at the sink is a property of the sink, not of Flink. KafkaSink only achieves it because it writes inside a transaction that is committed when the checkpoint closes, and that is why it requires a transactionalIdPrefix that is stable across restarts. FileSink works by the same logic: the files keep an “in progress” suffix until the checkpoint commits, which tends to scare anyone looking at the bucket midway and thinking nothing is being written. JdbcSink, on the other hand, has no distributed transaction, so the guarantee it offers depends on you writing with a primary key and letting repetition be harmless.

It is worth insisting on the versioning part, because it bites. Except for the file system connectors and datagen, which ship inside the distribution, every connector is a separate project, with its own repository, its own release and numbering that has no relation whatsoever to Flink’s. flink-connector-kafka is at 5.0.0 while Flink is at 2.3. The two numbers do not talk to each other, and there is no “the connector version that matches mine” by deduction.

What exists is a compatibility matrix, published on the downloads page, and checking that matrix is a mandatory step before choosing the Flink version for a new project. The reason is that the migration to the 2.x line was not simultaneous, and a good part of the ecosystem was left behind. A few examples of how this looked when I wrote this:

The Prometheus connector is the most eloquent case, and one that interests me closely, because it is the natural output for metrics derived from telemetry. It exists, it is official, and it writes through Prometheus’s remote write API. It is sink only, there is no corresponding source, which makes sense: Prometheus is where the time series ends, not where it comes from. Except that 1.0.0 is still from the Flink 1.x line and has not crossed over to 2.x. Anyone who wants that sink today chooses between staying on 1.20, which is LTS, or writing the integration by hand.

And notice that not even Kafka lists 2.3 in the table above. A one-release lag between the core and the connectors is the normal state of things, not an exception. If your pipeline depends on a specific connector, it is the connector that decides which Flink version you can run, and not the other way around.

Parallelism

Parallelism is the number of instances of each operator running at the same time. A job with parallelism 8 has eight copies of the source, eight of the map and eight of the sink, each processing a slice of the data and occupying a slot. Each of those copies is a subtask, which is why the web interface shows repeated metrics with indices from 0 to 7.

It can be set at three levels, from the broadest to the most specific. For the whole job, with env.setParallelism(8) in the code or parallelism.default in config.yaml. At deploy time, through the parallelism field of the FlinkDeployment, which is what appears in the manifest further down. And operator by operator, chaining .setParallelism(2) onto the call, which overrides the previous ones. The typical case for the override is a sink that talks to a database that cannot handle eighty simultaneous connections: the rest of the job stays wide and only the write is throttled on purpose.

For the initial value, a reasonable starting point is matching the parallelism to the number of partitions of the input Kafka topic. The reason not to go above that is specific and worth understanding, because the symptom is confusing: the leftover source subtasks receive no partition, sit idle and never emit a watermark. Since an operator’s watermark is the minimum across all its inputs, a stalled source holds back the clock of the whole job, and the result is a window that never closes and a job that looks stuck without a single error in the log. It can be worked around by declaring withIdleness() in the watermark strategy, which tells Flink to ignore the silent inputs, BUT IT IS SIMPLER NOT TO CREATE THE PROBLEM. Staying below the number of partitions also has a cost, a smaller one: you leave topic throughput on the table, because one subtask ends up reading more than one partition.

After that, tuning is by measurement, not by arithmetic. The two numbers that matter in the web interface are backpressure, which shows which operator is holding back the flow, and busy time, which shows the fraction of time each subtask was actually working instead of waiting. Raising the parallelism of an operator that spends the day idle improves nothing. I say more about this sizing in the CPU and slots section.

One piece is missing for rescaling to make sense, and it is internal: Flink does not distribute key by key. It runs each key through a hash and places it into a key group, and what gets distributed among the subtasks are the key groups, in contiguous ranges. The key group is the unit of state movement, that is, when you change the parallelism, what migrates from one machine to another are whole groups, never individual keys. You almost never touch this directly, but the term shows up in logs and in any conversation about rescaling.

The number of key groups that exist is exactly the maxParallelism. From there come the two properties that matter. It is the absolute ceiling of parallelism, because you cannot have more subtasks than groups to distribute among them. And it cannot change afterwards, because changing the number of groups changes which group each key belongs to, which invalidates all the state already saved.

The trap is in the default value. Without configuration, Flink computes that number from the initial parallelism: something around one and a half times it, rounded up, with a floor of 128 and a ceiling of 32768. A job that starts with parallelism 8 gets 128 and nobody notices, because nothing breaks. The problem arrives on the day that job needs parallelism 200, and the only way out is rebuilding the state from scratch.

Setting it by hand from the first run solves it, and env.setMaxParallelism(1024) is a comfortable value. What does not work is putting 32768 as a precaution: some state backends keep internal structures that grow with the number of key groups, so overdoing it costs performance. An extra care that comes for free is choosing a value divisible by the intended parallelism, because then the groups spread evenly. With 128 groups and parallelism 12, some subtasks get 11 and others 10, a small but avoidable asymmetry.

Redistribution and data skew

Between two operators there is always a decision about which next subtask each event goes to. Most of the time it is made by Flink without you asking, but you can choose, and the options are few:

broadcast deserves a note because it is the basis of an entire pattern: a small stream of rules is broadcast to all subtasks, which keep it in state, while the big stream of events passes partitioned by key. That is how you change a job’s logic without a redeploy.

It is also worth remembering that rebalance and rescale gained adaptive distribution in 2.3, looking at the real load of the destination instead of doing blind round-robin, and it is an option that needs to be switched on, as I describe in the post about Flink 2.3.

None of this solves the problem that hurts the most, which is data skew. Skew is when the distribution of keys is uneven: one customer who alone accounts for half the transactions, a product that went viral, a giant tenant in the middle of a thousand small ones. Since keyBy guarantees that the same key always goes to the same subtask, that subtask becomes the bottleneck, the others sit idle, and the whole job moves at the speed of the slowest one. Increasing parallelism does not help at all, because the heavy key keeps going to a single place.

The diagnosis is straightforward in the web interface, and that is why busy time matters so much: skew shows up as backpressure concentrated in one subtask and as one instance at 100% occupancy while its neighbors sit at 10%. If all of them were at 90%, the problem would be capacity, and the answer would be different.

There are three ways out, in order of effort. The first, and the most used, is aggregating in two phases: you append a random suffix to the key, something like customer-42#0 through customer-42#9, aggregate partially over that inflated key, and only then aggregate again over the real key. The heavy work gets split across ten subtasks, and the second stage receives only ten partial results per key instead of millions of events. The second, when you are in SQL, is letting the engine do it: enabling mini-batch and the two-phase aggregation optimization solves the common case without rewriting a single query, and there is even a specific option to split DISTINCT aggregations, which is the automatic version of the same trick. The third is accepting that the whale key is a case apart and handling it outside the normal flow, with a dedicated pipeline (a classic).

Operator chaining

The graph you write is not exactly the graph that runs. Before executing, Flink merges adjacent operators into a single task, and that is called operator chaining. A sequence like map, filter and flatMap becomes a single thing, executed in the same thread.

The merge only happens under three conditions: the link between the operators needs to be forward, they need to have the same parallelism and they need to be in the same slot sharing group. A keyBy or a rebalance breaks the chain, because in those cases the record really does need to leave one instance and go to another.

The gain is bigger than it looks at first glance. Inside a chain, passing a record from one operator to the next is a method call, and the object stays the same in memory. Between two tasks, the record is serialized, goes into a network buffer and is deserialized on the other side, and that happens even when both are inside the same TaskManager. Chaining eliminates that whole cost.

It is also the explanation for a common confusion in the web interface: the boxes in the graph do not correspond to your operators, they correspond to the chains. You write six operators and see two boxes, each listing inside it the names it absorbed.

The trap appears when investigating a problem. Metrics and backpressure are measured per task, not per operator, so an entire chain reports a single number, and you cannot tell which of the merged operators is holding back the flow. To find out, you can break the chain on purpose: .disableChaining() isolates an operator on both sides, and .startNewChain() starts a new chain from it. This is a diagnostic tool, not a setting to leave on. The serialization cost you reintroduced is real, so it is worth reverting after measuring.

Rich functions

Every function interface has a “rich” version: MapFunction has RichMapFunction, FilterFunction has RichFilterFunction, and so on. The difference is that the rich version adds a lifecycle and context, and that is what separates a function that merely transforms a record from a function that needs external resources or memory.

open() is called once per parallel instance, before the first event arrives. It is the place to open a connection to a database, load an enrichment cache, compile a regular expression and, above all, register the state descriptors. Note the “per parallel instance”: with parallelism 8 it runs eight times, once in each subtask, and what it creates is not shared among them. close() is its counterpart, called on shutdown, and serves to return what was opened.

getRuntimeContext() is the entry point to what the runtime knows. Through it you reach keyed state, the custom metrics you want to expose, the index of the current subtask and the operator’s parallelism.

If this sounds familiar when you get to the state section, it is because the example there already uses both: AnomalyDetector overrides open() and calls getRuntimeContext().getState(...) to create the ValueState. It does not have “Rich” in its name, but KeyedProcessFunction extends AbstractRichFunction and inherits the same lifecycle. Process functions come rich out of the box.

A version note that saves time: up to 1.x the signature was open(Configuration parameters). In 2.x it was removed, and the method now receives an OpenContext. An example copied from an old tutorial does not compile because of this.

Time: the part that breaks intuition

This was the section that most made me stop and reread. In streaming there are, at a minimum, two different times:

• Event time: when the event actually happened. It is recorded inside the event itself.
• Processing time: when Flink saw the event. It is the machine’s clock.

It seems like a detail, but it is not. Someone’s phone lost signal in the subway and synced twenty minutes later. A broker had a latency spike. One Kafka partition fell behind the others. In all those cases, events arrive out of order and late.

If you group by processing time, the result depends on when the job ran. Reprocessing the same day tomorrow gives a different number. If you group by event time, the result is always the same, regardless of when the processing happened. That reproducibility is the reason event time is the default in almost every serious pipeline.

But that creates a new problem: if events arrive out of order, how do you know when a time window can be closed? No event ever arrives saying “that’s all, you can count now”.

Watermarks

Flink’s answer is watermarks. A watermark is a marker that travels along with the stream carrying a claim:

“I believe I will no longer receive events with a timestamp earlier than T.”

It is a bet, not a certainty. And it is precisely because it is a bet that it is useful: it turns an impossible question (“has everything arrived?”) into a configurable decision (“how much delay do I tolerate?”).

WatermarkStrategy
    .<Event>forBoundedOutOfOrderness(Duration.ofSeconds(10))
    .withTimestampAssigner((event, ts) -> event.getTimestamp());

This code says: tolerate up to 10 seconds of disorder. If the largest timestamp seen so far is 12:00:30, the watermark is 12:00:20, and any window ending before that can be closed and emitted.

The trade-off shows up immediately:

• Aggressive watermark (little delay tolerated) → fast result, more events lost for arriving late.
• Conservative watermark (a lot of delay tolerated) → correct result, but you wait longer to see anything.

There is no right value. There is the value that matches your business, and that is a product decision disguised as a technical setting.

For events that arrive after the watermark has advanced, Flink offers specific mechanisms for handling late data. allowedLateness keeps the window open for an additional period, allowing late events to be processed and already-produced results to be updated. Another alternative is to use a side output, routing the events that exceeded the lateness limit to a separate stream. That mechanism can be useful for auditing, data quality metrics or for identifying problems in the watermark strategy configuration.

Windows

Since the stream is infinite, aggregations only make sense within slices.

• Tumbling: fixed windows that do not overlap. “Every 5 minutes, how many orders?”
• Sliding: fixed windows that do overlap. “In the last 10 minutes, updating every 1 minute.” The same event falls into several windows.
• Session: windows defined by inactivity. “Group a user’s activity until they stay idle for 30 minutes.” The window size is given by the data, not by you.

stream
    .keyBy(Event::getUserId)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .aggregate(new EventCounter());

A detail I had not understood right away: prefer reduce/aggregate over process whenever you can. process receives all the events of the window at once, which means keeping all of them in state until it closes. aggregate keeps only the accumulator. In large windows, that difference is the difference between a stable job and a job that dies of OOM.

ProcessFunction

The ready-made operators cover a lot, but they impose a shape. map turns one event into another, window groups by a time slice, aggregate accumulates. When the logic does not fit any of those molds, the path is to go down a level and write a ProcessFunction, which is the lowest-level and most powerful operator in the DataStream API. It is Flink’s Swiss army knife, and it is no exaggeration to say that almost all truly complicated business logic ends up inside one.

What it gives you, on every event, are three things the ready-made operators do not give together: direct access to the key’s state, the ability to register timers and the ability to emit to side outputs. All three arrive through the same place, that ctx parameter in the processElement signature.

The timer is the part that changes what can be built, because it is what allows reacting to the passage of time, and not only to the arrival of an event. You register a future instant and Flink calls your onTimer when it arrives:

@Override
public void processElement(Event event, Context ctx, Collector<Alert> out) {
    ctx.timerService().registerEventTimeTimer(event.getTimestamp() + 600_000);
}

@Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<Alert> out) {
    // ten minutes of event time have passed since that event
}

Two properties of this mechanism matter. The first is that an event time timer does not fire when the machine’s clock gets there, but when the watermark passes that instant, which ties timers to the same notion of time as windows: if the watermark does not advance, the timer does not fire, and we are back to the idle source problem I described earlier. The second is that timers are state, they belong to a key and go into the checkpoint. A job that restarts does not lose its pending timers, and that is what makes it viable to write “if nothing happens in the next thirty minutes, alert” and trust the sentence.

The side output is the second door. It already appeared here as the destination for late events, but it serves any fork: records that failed validation, events that triggered a rule, samples for auditing. You declare an OutputTag, call ctx.output(tag, value) and retrieve that flow later with getSideOutput(tag). The advantage over a duplicated filter is that the event is examined only once, and the advantage over throwing an exception is that the job does not die because of a malformed record.

There are a few variants, and choosing the wrong one is a common mistake. KeyedProcessFunction is the one that comes after a keyBy and the only one with keyed state and timers. The plain ProcessFunction works over a non-partitioned stream and has neither. CoProcessFunction receives two streams and is the path to join a flow of events with a flow of rules or reference data. And ProcessWindowFunction is the one that runs when a window closes, with all of its events in hand.

The price of going down to that level is that Flink stops taking care of things for you. A window knows when to end and throws away what it accumulated; a state you created inside a ProcessFunction stays there forever, for every key that ever appeared, until you explicitly delete it or configure TTL. It is the same freedom and the same risk of any low-level abstraction, and it is the reason the recommendation remains to use the ready-made operator whenever it fits.

State

I started studying Flink by writing, before it, my own Kafka consumer. The idea was to answer a single question: what does a distributed processing framework solve beyond reading events from a topic?

The answer came quickly, and it is a list. State management, parallelism, partitioning, fault tolerance, recovery after a crash, event time, watermarks, coordination between processing stages. In the manual consumer, every one of those items becomes your code. In Flink, almost all of them are part of the runtime, and what is left for the application is the processing logic.

Any more interesting stream processing needs to keep some kind of memory, whether to count events, deduplicate, detect patterns, correlate data or compare previous values. In a traditional consumer, that responsibility ends up much closer to the application. In Flink, state is managed by the runtime and can be distributed, partitioned by key, stored persistently and recovered after a failure through the checkpoint and recovery mechanisms. That changes quite a lot how you build a streaming application, especially when we start thinking about scale, failures and continuous processing.

Flink offers state as a first-class primitive, with types like ValueState, ListState and MapState, always associated with a key after the keyBy.

What the hell is a first-class primitive??

It comes from programming language theory. Something is first-class when the system treats it as a full citizen, with all rights, instead of an appendix you have to manage from the outside.. it means you do not need a database. Here is the part that really matters. The obvious alternative would be to keep that accumulation in an external Redis or Postgres. Flink does it differently: the state lives inside the operator itself, in the TaskManager process, managed by the framework. That brings three consequences.

First, it is local. Reading state is a memory access or an access to a RocksDB on the machine’s local disk, not a network call. That is the difference between microseconds and milliseconds per event, and the reason Flink can process millions of events per second with logic that depends on history.

Second, it goes into checkpoints. Periodically Flink takes a consistent snapshot of all the state of all operators and writes it to durable storage (S3, HDFS). If a TaskManager dies, the job restarts from the last checkpoint with the state exactly as it was, and the Kafka offsets come back with it. That is what sustains the exactly-once semantics we mentioned earlier. With an external Redis, you would have to solve that consistency between “what I have already consumed from Kafka” and “what I have already written to Redis” by hand, and it is notoriously hard.

Third, it is resizable. When you increase the job’s parallelism, Flink redistributes the state among the new instances automatically, from a savepoint.

public class AnomalyDetector extends KeyedProcessFunction<String, Event, Alert> {

    private transient ValueState<Double> lastValue;

    @Override
    public void open(OpenContext openContext) {
        lastValue = getRuntimeContext().getState(
            new ValueStateDescriptor<>("last-value", Double.class));
    }

    @Override
    public void processElement(Event event, Context ctx, Collector<Alert> out)
            throws Exception {
        Double previous = lastValue.value();
        if (previous != null && event.getValue() > previous * 3) {
            out.collect(new Alert(event, previous));
        }
        lastValue.update(event.getValue());
    }
}

That state is partitioned by key: each user has their own lastValue, and the operator only sees the key of the current event. You can also configure TTL, which avoids the classic problem of state that only grows until it blows up the disk.

Where that state is kept is a choice, and there are three options in the 2.x line:

The default, if you configure nothing, is HashMapStateBackend. That tends to surprise people who read that “Flink uses RocksDB”, and the switch is one line in state.backend.type, which accepts hashmap, rocksdb or forst.

The rule of thumb I took from this has three cases. State of a few gigabytes with critical latency calls for HashMap, because the absence of serialization is exactly what you are buying. Large state, from tens of gigabytes to terabytes, calls for RocksDB with incremental checkpoints, and it is worth noting that it is the only one of the two classics that offers this: with HashMap, every checkpoint copies the entire state. And huge state in the cloud, with aggressive autoscaling, is the case for evaluating ForSt, which is the bet of the 2.x series and which I write more about at the end.

Checkpoints and savepoints

State is only reliable because there is a fault tolerance mechanism behind it.

Periodically, the JobManager injects checkpoint barriers into the sources. Those barriers flow through the graph together with the data; when an operator receives the barrier, it persists its state to durable storage (S3, HDFS, whatever) and passes the barrier along. When all operators finish, the checkpoint is complete.

The algorithm is a variation of Chandy-Lamport for distributed snapshots. I found it beautiful to discover that something so practical rests on a paper from 1985.

On failure, Flink restores the last complete checkpoint, repositions the source offsets and resumes. From that comes the famous exactly-once, which I misunderstood for quite a while. It does not mean every event is physically processed a single time. On recovery, events are indeed reprocessed. It means the effect on the state is as if each event had been processed only once. It is a guarantee about the result, not about the execution.

Extending that guarantee all the way to the sink is another matter, and depends on the destination supporting transactional or idempotent writes (Kafka with transactions, for example).

The difference between checkpoint and savepoint also took a while to settle:

A savepoint is what allows you to stop a job, bring up a new version of the code and continue exactly where it left off, with the state intact. In practice, that is what makes a streaming job something you can operate for years.

Flink SQL

Not everything needs Java. Flink exposes a Table API and SQL over the same engine, and the same query runs both over finite data and over an infinite stream:

SELECT
    user_id,
    TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
    COUNT(*) AS total
FROM orders
GROUP BY
    user_id,
    TUMBLE(event_time, INTERVAL '5' MINUTE);

The concept underpinning this is the stream/table duality: a stream is the changelog of a table, and a table is the accumulated state of a stream. They are the same information seen from two angles.

This stops being abstract as soon as you run a GROUP BY in the SQL client and look at the op column:

+----+------+-------+
| op | name | total |
+----+------+-------+
| +I |    a |     1 |
| +I |    b |     1 |
| -U |    a |     1 |
| +U |    a |     2 |
+----+------+-------+

a comes in with 1, is retracted (-U) and re-emitted with 2 (+U). It is not a table being filled in bit by bit, it is its changelog passing on the screen. The result of a streaming query is not a final value, it is a sequence of corrections over the previous value.

There are four row kinds in total: +I for insertion, -U and +U for the before and after of an update, and -D for deletion. And it is the combination of kinds a query produces that defines what type of stream you are dealing with, which in practice decides where it can be written:

The difference between retract and upsert is subtle and matters when choosing the destination. In retract mode, Flink emits two rows for each correction, and the receiver needs to know how to subtract the old one before adding the new one. In upsert mode, it emits only the new row, and the receiver overwrites what it had for that key. The second is cheaper and is what relational databases and search indexes expect, but it is only possible when there is a key that identifies the row.

That leads to the classic trap, which can be predicted before falling into it. You write a continuous GROUP BY and point the result at an ordinary Kafka topic. The standard Kafka connector is append-only, and the result of your query is a flow of corrections, not of insertions. The job does not even start: the planner rejects the query with a message like “table sink doesn’t support consuming update changes which is produced by node GroupAggregate”. Failing at planning time is good news, because the silent way of getting this wrong would be writing the retractions as if they were normal events and having a consumer downstream adding everything up twice.

There are three ways out, and the choice says more about the problem than about Flink. The first is using a sink that accepts updates, like upsert-kafka or JDBC with a declared primary key. The second is making the result stop changing, by grouping by window instead of grouping continuously: when the window closes, the value is final and becomes a single insertion. The third, available from 2.3 onwards, is explicitly converting between the forms with FROM_CHANGELOG and TO_CHANGELOG, which is what I describe in the post about Flink 2.3.

For people coming from data, this is the fastest entry point, you can get far without writing a line of Java.

In praxi (“in practice”)

Bringing up a local cluster is simpler than I imagined. I started with two loose docker run commands, but switched to a docker-compose.yml as soon as I wanted more than one TaskManager:

name: flink-lab

services:
  jobmanager:
    image: flink:2.3.0-java21
    container_name: jobmanager
    command: jobmanager
    ports:
      - "8081:8081"
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        rest.address: 0.0.0.0
        rest.bind-address: 0.0.0.0
        jobmanager.memory.process.size: 1024m
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:8081/overview || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 20
      start_period: 10s

  taskmanager:
    image: flink:2.3.0-java21
    command: taskmanager
    depends_on:
      jobmanager:
        condition: service_healthy
    deploy:
      replicas: 2
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2
        taskmanager.memory.process.size: 1728m

One docker compose up -d and the Web UI answers at localhost:8081.

Three choices there are not details. Two TaskManager replicas with two slots each give four slots, and that is what makes parallelism stop being always 1: without spare slots, every job graph shows up in the UI as a straight line. The healthcheck combined with depends_on: service_healthy exists because, without it, the TaskManager starts before the JobManager is up, fails to register and fills the log with errors before working on the second try. And the memory limits are explicit because Flink’s default is generous: with these values the whole cluster stays around 4.5 GB and fits on any machine.

Before submitting anything, the REST API tells you whether the cluster is whole more directly than the UI:

curl -s localhost:8081/overview

{"taskmanagers": 2, "slots-total": 4, "slots-available": 4, "flink-version": "2.3.0"}

The two registered TaskManagers and the four free slots are the confirmation that the cluster formed. The image still ships example jobs, so you can see a running graph without writing a line:

docker exec jobmanager ./bin/flink run -d \
  /opt/flink/examples/streaming/TopSpeedWindowing.jar

With it running, slots-available drops to 3 and the UI finally has something to show.

The Web UI was where I learned the most. It shows the job graph, the real parallelism of each operator, the checkpoint history (duration and size) and, above all, backpressure: when an operator cannot keep up with the pace of the previous one and the pressure propagates backwards all the way to the source. Watching that happen in real time teaches more than any diagram.

For SQL, the shortest path is the interactive client:

docker exec -it jobmanager ./bin/sql-client.sh

Deployment

The docker-compose.yml above is a lab. It brings up a cluster, but hides almost all the decisions that appear when the job needs to run for months without anyone watching. This was the part that most forced me to read the manual, because here no intuition saves you: it is arithmetic, it is container limits and it is choosing where the state will end up.

Execution modes

Before choosing where to run, you need to choose how. Flink has two living modes today, and the difference between them is the cluster’s lifecycle.

In session mode, the cluster comes up first and stays up waiting. You submit several jobs to the same JobManager, and all of them compete for the same TaskManagers. It is what my compose does, and it is the right mode for exploration, for the SQL client and for short jobs that do not justify the cost of bringing up a new cluster. The price is the absence of isolation: a job that blows up memory takes down the TaskManager, and with it go the tasks of all the other jobs that had slots in that process.

In application mode, each application gets its own dedicated cluster, which is born with it and dies with it. The most important difference is subtle and lies in main(): instead of running on the machine of whoever submits, it runs inside the JobManager. That takes away from the client the responsibility of downloading dependencies and assembling the job graph, and it is what makes the mode suitable for automation, because there is no longer an external client process that needs to stay alive. For production, it is the default answer.

There was also a third mode, per-job, which also gave one cluster per job but kept main() on the client. It was discontinued and is no longer part of the 2.x line. If you find -t yarn-per-job in some tutorial, it is material from an earlier version.

Orthogonal to this is the resource manager: standalone (you bring up the processes yourself, which is the Docker Compose case), Kubernetes or YARN. The combination I see recommended most often today is application mode on Kubernetes.

Kubernetes

On Kubernetes there are three paths, and it is worth understanding the difference because they solve distinct problems.

The first is standalone on Kubernetes: you write two Deployments, one for the JobManager and another for the TaskManager, a Service and a ConfigMap with the flink-conf.yaml. Flink does not know it is on Kubernetes, it only sees processes that register. It works, and it is transparent, but scaling is on you: changing parallelism means changing replicas by hand, and upgrading with a savepoint is a sequence of commands someone needs to remember to run in the right order.

The second is native Kubernetes, in which the JobManager itself talks to the Kubernetes API and creates the TaskManager pods needed to satisfy the requested parallelism. You submit with flink run-application -t kubernetes-application and write no TaskManager manifest at all. The cluster sizes itself according to the job, and not the other way around.

The third, and the one I would use, is the Flink Kubernetes Operator. It installs CRDs and turns the job into a declarative cluster object:

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: otlp-processor
spec:
  image: registry.example.com/otlp-processor:1.4.0
  flinkVersion: v2_0
  serviceAccount: flink
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "4"
    state.backend.type: rocksdb
    state.checkpoints.dir: s3://flink-state/otlp-processor/checkpoints
    state.savepoints.dir: s3://flink-state/otlp-processor/savepoints
    execution.checkpointing.interval: "60s"
    high-availability.type: kubernetes
    high-availability.storageDir: s3://flink-state/otlp-processor/ha
  jobManager:
    resource:
      cpu: 1
      memory: 2048m
  taskManager:
    replicas: 3
    resource:
      cpu: 4
      memory: 8192m
  job:
    jarURI: local:///opt/flink/usrlib/otlp-processor.jar
    parallelism: 12
    upgradeMode: savepoint

What this manifest buys is operations. upgradeMode: savepoint means that, when I change the image tag and apply again, the operator triggers a savepoint, stops the job, brings up the new version and restores the state from it, without anyone needing to run flink stop --savepointPath by hand at three in the morning. It also takes care of rollback when the new job does not stabilize, and brings an autoscaler that watches the real backpressure of the operators and adjusts the parallelism, which is quite different from an HPA looking at the pod’s CPU usage.

Two details that are not optional. The first is the serviceAccount: the JobManager needs RBAC permission to create pods and to read and write ConfigMaps, because it is in a ConfigMap that native high availability keeps who the leader is and what the last checkpoint is. The second is the high-availability.storageDir: the ConfigMap keeps only the pointer, the real metadata goes to durable storage. Without that directory configured, a JobManager that dies comes back not knowing where to resume from, and the entire state is lost despite the checkpoints existing.

Memory

This is the arithmetic that takes the most work, because Flink does not have one memory number, it has a tree of them. The value you configure is taskmanager.memory.process.size, and it is the total the process can occupy in the operating system. Everything else is subdivision.

Inside it, Flink first separates what belongs to the JVM and not to Flink: the metaspace (taskmanager.memory.jvm-metaspace.size, 256m by default) and an overhead for thread stacks, internal buffers and native code (taskmanager.memory.jvm-overhead.fraction, 10% of the process size). What remains is Flink’s memory proper, and it splits into four parts:

• The task heap, where your objects live, and the state when the backend is the heap one.
• Managed memory, which is off-heap and is where RocksDB allocates its block caches and write buffers. In a batch job, it is also where sort and hash join happen.
• Network memory, which is the buffers for exchanging data between tasks.
• A slice reserved for the framework, which you rarely touch.

The rule that matters above all others: the process.size has to fit inside the container’s memory limit. If you give the pod an 8Gi limit and configure process.size: 8192m, the pod will be killed by OOMKill, because the cgroup also counts file pages, native RocksDB allocations outside what Flink accounts for and what the kernel itself uses. Leaving a 10 to 15% margin solves it. For an 8Gi limit, I would configure around 7000m and sleep better.

The second decision is how to split. If the state is large and the backend is RocksDB, managed memory is what makes the difference, because it determines the size of the cache before access turns into disk reads. A reasonable starting split is 40% managed memory and the rest between heap and network. If the state is small and fits in heap, the opposite path: taskmanager.memory.managed.fraction: 0.1 and everything else to the heap.

Network memory is the most surprising one. It is 10% of Flink’s memory by default, capped at 1 GB, and consumption grows with the number of connections between tasks, which in turn grows with the square of the parallelism in keyBy-style exchanges. A job with parallelism 4 that moves to 200 can start failing with an insufficient buffers error without a single line of code having changed. It is one of the few cases where increasing parallelism makes things worse before it makes them better.

CPU and slots

A slot is not a CPU. A slot is a slice of memory and a place in the scheduling, but the threads of all the slots in a TaskManager compete for the same cores of the JVM. There is no CPU isolation between slots.

That is why the starting rule is simple: taskmanager.numberOfTaskSlots equal to the number of cores the container has as its limit. Four cores, four slots. Going much above that does not increase throughput, it only increases context switching and the pause time the garbage collector has to manage.

The total parallelism of the job is then the number of TaskManagers times the slots of each one, and that multiplication is the number that appears in the manifest. Three TaskManagers of four slots give twelve, which is the parallelism of the example above.

To get to the right number of TaskManagers, I would not try to calculate from first principles. The path that makes sense is measuring: bring up the job with low parallelism, inject a known load and look at numRecordsInPerSecond per subtask in the UI or in Prometheus. That gives a throughput per slot. Divide the expected peak by that number, add a margin of about 30% for spikes and for reprocessing (because after a crash the job needs to run faster than production to catch up to the present) and the result is how many slots are needed. A filter-and-projection job can do hundreds of thousands of events per second in one slot; a job with a large window, state in RocksDB and JSON deserialization can do a few thousand. The difference between those two cases is orders of magnitude, and that is exactly why you cannot guess.

It is worth remembering that changing the parallelism of a stateful job later is not free: the state is redistributed among the new subtasks on restore, and the ceiling remains the maxParallelism fixed on the first run, for the reason I described in the parallelism section.

Local disk

With RocksDB, the working state is not in memory, it is in files on the TaskManager’s local disk, pointed to by io.tmp.dirs. That is what allows having state larger than RAM, and it is also what makes the type of disk matter.

It needs to be local SSD, not a network volume. RocksDB is an LSM tree: it writes to sorted files and then compacts those files in the background, rewriting data that was already there. That write amplification, added to random reads on every state access, is exactly the I/O pattern a network disk serves badly. I have seen the same job go from stable to unusable just by switching the volume type.

For sizing, the arithmetic starts from the logical state: number of active keys times the state size per key, divided by the parallelism, gives the state per TaskManager. After that, multiply. Two to three times is a reasonable margin, because compaction needs space to write the new files before deleting the old ones, and because incremental checkpoints keep files referenced by earlier snapshots that have not yet been discarded. Ten million keys with 1 KB of state each give 10 GB of logical state; across three TaskManagers, that is something between 7 and 10 GB of disk on each.

On Kubernetes, that means choosing between emptyDir and a persistent volume. With emptyDir, the disk dies with the pod, and a restart forces the TaskManager to download the entire state from remote storage before it resumes processing, which for large states is the difference between seconds and many minutes of recovery. With a persistent volume and state.backend.local-recovery: true, the local copy survives the restart and recovery reads from disk instead of the network. It is more complexity in exchange for recovery time, and the choice depends on how much the delay costs.

S3 and the remote file system

Here is the confusion I myself had before studying: S3 is not where the state lives, it is where copies of it are kept. The hot state is in the heap or on the TaskManager’s local disk, and that is where it is read and written on every event. S3 enters at four moments, and all of them are periodic or exceptional.

The first is checkpoints. At every configured interval, each operator writes its state to state.checkpoints.dir. That is what survives the simultaneous loss of all pods.

The second is savepoints, in state.savepoints.dir, which is the same mechanism with a different lifecycle: they are yours, they are not deleted by Flink, and it is from them that the upgrade without losing state comes.

The third is the high availability metadata, in high-availability.storageDir. The Kubernetes ConfigMap is too small to hold the job graph and the full pointers, so it keeps only the reference and the content lives in remote storage.

The fourth, when it exists, is the data itself: a source that reads Parquet from a bucket, or a sink that writes results partitioned by hour.

The reason it is S3, and not a shared disk, is that durability needs to be independent of the cluster. A checkpoint that is on the same hardware as the job does not protect against the loss of that hardware. Object storage is cheap, replicated and keeps existing after the whole cluster stops existing, which is precisely the property that makes a streaming job recoverable. GCS, Azure Blob and HDFS occupy the same place in the architecture, with a different letter in the URI scheme.

The configuration has a gotcha that costs time. Flink ships two S3 implementations, and they are not interchangeable. flink-s3-fs-presto is the one indicated for checkpoints, because it is fast with small files and does not depend on renaming objects, an operation S3 does not have atomically and emulates with a copy followed by a delete. flink-s3-fs-hadoop is the one that supports recoverable writes, which is what the file sink needs to do multipart uploads and commit only at the checkpoint. Many people install both and use s3p:// for state and s3a:// for data. Both are plugins and need to go to /opt/flink/plugins/, each in its own directory, and not to lib/. In lib/ they load in the main classloader and conflict with each other.

The last point is about the checkpoint interval, which looks like a latency setting and is also a cost calculation. Every checkpoint writes at least one file per stateful subtask. A job with parallelism 200 and a checkpoint every 10 seconds produces on the order of seventy thousand objects per hour, and S3 charges per request besides rate-limiting per prefix. That shows up as rising checkpoint latency and, at the limit, as checkpoints expiring by timeout. Two defenses help: raising state.storage.fs.memory-threshold so that small states travel inside the metadata file instead of becoming separate objects, and accepting a longer interval. One minute between checkpoints is a common value, and what it costs is only the amount of work that needs to be redone after a failure.

The ecosystem around it

Flink rarely appears alone. It is a processing engine, and an engine needs something to feed it, somewhere to deposit and people who read what it produced. A good part of the work of assembling a pipeline is choosing those pieces, so it is worth mapping the ones that most often appear next to it and, above all, why they appear.

Three of those choices have a detail that only appears later, and it is worth anticipating.

The first is the small files problem. The Iceberg sink commits what it wrote when the checkpoint closes, so the checkpoint interval stops being only a recovery decision and starts determining how many files and how many snapshots the table gains per hour. A checkpoint every ten seconds in a job with high parallelism produces a table with thousands of tiny files per day, which is the worst possible format for whoever reads it later. The solution is not avoiding Iceberg, it is accepting that compaction and snapshot expiration are recurring work, and it is precisely that work that usually justifies an Airflow next to a pipeline that, on paper, had nothing scheduled.

The second is the catalog, which is almost always remembered late. It is easy to think about the table format and forget that someone needs to know the table exists. Without a configured catalog, Flink SQL’s CREATE TABLE lives only in the session and disappears when the client closes, and the Trino on the other side sees nothing of what was created. Choosing the catalog is what makes Flink and the other engines talk about the same table, instead of the same files by coincidence.

The third is the reason a feature store exists, which took me a while to understand. The problem it solves is not storage, it is consistency: if the feature “average purchases in the last 30 minutes” is computed by a Flink job at inference and by a SQL query at training, the two will disagree in some detail, and the model starts seeing in production data different from what it saw while learning. Keeping a single definition, fed by the same job, is the point. And it is worth noting that part of this is migrating into Flink itself with the model functions of 2.1 and 2.2, which I describe in the post about Flink 2.3.

Complexities

• Fine-tuning RocksDB and the behavior of state when it grows far beyond memory.
• Join strategies between streams, mainly interval joins and temporal joins.
• The CEP library, mentioned up in the use cases and not yet tested by me.
• How all of this really behaves in production under irregular load, the part no documentation teaches.

What stayed with me

I ended the week with an impression I did not expect to have: the hard part of streaming is not distributing the processing. Flink solves that. The hard part is time: accepting that data arrives out of order, that “complete” is a bet and not a fact, and that almost every technical decision here is, deep down, a choice between latency and correctness.

Writing this forced me to admit three or four things I thought I knew and did not. It was worth the effort.

It remains to say why I went after this, because it is not Flink for Flink’s sake. What I want is to process OTLP, the format the OpenTelemetry Collector speaks, and in practice that means the four signals: logs, spans, metrics and profiles.

The Collector receives, transforms and forwards very well, but it is built for data to pass through. The moment the question needs memory between events, event-time windows or correlating signals that arrive separately and out of order, it falls exactly into what this text describes: keyBy, keyed state and watermarks. That fit is what I want to test in the next round.

Next week on the schedule: reviewing the Data Engineer roadmap. I keep writing while I learn.

What is the Republic? It is one of Plato’s most important and best-known works, probably written in the 4th century BC, around 375 BC. The work is built mainly in the form of dialogues, with Socrates as the central character.

One of the great problems guiding the discussion is an apparently simple yet quite uncomfortable question: is it worth being just? What is justice? Is it better to be just for its own sake, or only for the advantages and rewards it may bring? In other words: is it worth acting rightly when injustice could bring more benefits?

From these questions, Plato develops a long inquiry into justice, education, the organization of the city, the nature of the soul and the way we ought to live.

It is in this context that the Myth of Er appears, closing the work. The myth tells the experience of Er, a man who supposedly died in battle and returned to life to tell what he saw after death. His account deals with the fate of souls, the consequences of their choices and the possibility of choosing a new life.

Being at the end of the Republic, the Myth of Er works as a symbolic conclusion of the work and takes up one of its fundamental questions: what are the consequences of our choices, and what kind of life should we choose? It should not be understood simply as Plato’s “final answer”, but as a narrative that closes and expands the philosophical inquiry developed throughout the dialogue.

Who tells the story? Socrates, who was Plato’s teacher and is the main character in almost all his books. It is through his mouth that the myth is narrated.

The setting

Er is a warrior who dies in battle. His body lies for days on the field without rotting and, when it is about to be cremated, he wakes up and tells everything he saw on the other side.

Basically: the first near-death experience report in the history of philosophy.

What Er describes is the process by which souls return to the sensible world. Translating the technical term: the “sensible world” is simply the world we see, touch, smell, the material world, THE ONE HERE. Plato believed the soul exists before and after the body, and that it goes and comes back. What Er saw was precisely the boarding desk of that return. And there are people working there. The Moirai, three figures from Greek mythology who control each person’s fate, represented as spinners (women who spin, that is, produce thread). The idea is that your life is a thread, and they take care of that thread:

• Clotho: the one who spins, that is, produces the thread of your life.
• Lachesis: the one who measures and distributes, draws lots and organizes destinies.
• Atropos: the one who cuts the thread. She is death. Her name literally means “the inflexible”, the one who does not turn back.

Keep that detail, because it matters: the Moirai are there, fate exists in the story. Plato is not denying fate. He is going to do something smarter than that.

The rules of the game

Now comes the good part. The rebirth of souls follows very specific rules.. and each one carries a message.

1. There is a lottery

When the souls are ready to be reborn, they take part in a lottery. But the lottery does not define the life you will have. It only defines the order in which each soul will choose. It is a queue ticket, not a prize. You do not draw what you will take; you draw when you will get to pick.

2. There is enough life for everyone

And here is the central point of the myth: there are more life options than souls. Chance takes part in the process. The lottery exists and determines the order in which each soul will choose, but the choice of the life each one will lead is, ultimately, its own. And, along with the choice, comes the responsibility for it. That idea hit me in an unexpected way. It was the kind of passage that made me stop reading and stare at the ceiling for a good while.

3. The gods are not to blame

Plato hammers on this with full force. The sentence that appears in the myth is practically a verdict: the blame belongs to the one who chooses, and the divinity is blameless. Nobody shoved anything down anybody’s throat. Each soul chooses its path according to the wisdom it has, or according to the lack of it. In the Greek context, that was quite a provocation. The culture of the time was full of stories in which the gods ruined people’s lives out of whim, jealousy or revenge. Plato is cutting that excuse at the root.

The menu of lives

And what was available to choose from? Everything. Rich lives, poor lives, lives of tyrants, lives of famous people, lives of nobodies, simple lives, even the lives of animals. A full buffet of possible existences. But that is where things get really complicated. Because the life that looks good is not always good. The life of a tyrant shines from afar: power, money, everyone obeying you. Up close, it is a machine for producing injustice and, according to Plato, whoever produces injustice ends up destroying themselves from within. There is a great scene in the myth: a soul throws itself at the life of a tyrant, grabs it with everything it has, and only afterwards realizes what came bundled in the package. Then it leaves lamenting and blaming the gods. But it chose for itself. It just did not look properly. So the real challenge was never choosing the flashiest life. The challenge was having enough self-knowledge to choose a life that led to virtue and not to injustice.

Quickly on those two terms, because they are the heart of the text:

• Virtue, for Plato, is not being “nice” or following rules of moral etiquette. It is the soul working well, in order, with reason in charge, like a tuned instrument.
• Injustice is the opposite: the soul in disarray, ungoverned, with the impulses in command. For Plato, this is a kind of disease of the soul. And that is why being unjust is bad for you, even if nobody finds out and even if you profit from it.

And there is a cruel detail: character

Character, one’s way of being, the moral structure of a person, is very hard to change after the choice is made. In other words: you do not only choose the circumstances of your life. You also choose, bundled and built in, the kind of person you will be within it. It all comes in the same package. That is why the myth insists so much that the most important skill in the world is being able to foresee the good and the evil a life can bring. Knowing how to look at an option and see not the shine on the surface, but what it will do to you on the inside, in the long run. Without that, you are betting in the dark with your own destiny.

And why don’t we remember any of this?

Good question, and the myth answers it. Before reincarnating, the souls cross an arid plain and drink the water of the River of Forgetfulness (in Greek, Lethe, a word that literally means “forgetting”). One sip and done: everything is erased. That is why nobody arrives here remembering their own choice. We are born without the manual, without the receipt, without the memory of having signed at the bottom. Which is a bit mean, I admit. But it is also exactly what makes life a real life, and not the execution of a script you have already read before.

What Plato really meant by all this

1. It is a frontal attack on fatalism

Fatalism is the belief that everything is already written and nothing you do matters, “if it is meant to be, it will be”. Its cousin, determinism, is the idea that everything you do was already caused by things outside your control: genetics, upbringing, society, chance. Notice what Plato does: he does not deny that fate exists. The Moirai are there, weaving. The lottery exists. Chance exists and really interferes. He could simply have said “fate does not exist”, but he did not. What he does is more sophisticated: he shows that, even within fate, freedom remains. You do not choose your position in the queue. You choose what you do with it. And since there is always a good option left on the table, the excuse “I had no alternative” simply does not hold.

That is the central thesis: the human being is free and, being free, is responsible for the path they walk.

2. It is a message about whose fault it is

The entire structure of the myth serves to close the emergency exits we use to escape responsibility.

Blame the gods?? Denied.. the divinity is innocent. Blame the lottery?? Denied.. there was good life left over. Blame the lack of options?? Denied.. the menu was huge. Blame whoever chose before you?? Denied.. ask Odysseus.

What is left?? You =)))))

3. The “choice” did not happen in an afterlife, it is happening now

This is the most important point, and it is where the myth stops being a story about reincarnation and becomes a story about today. Remember that character comes built into the choice and is almost impossible to change afterwards? Remember that the soul that grabbed the tyrant’s life did so on impulse, without examining? Well: that is an image of what we do all the time.

With every decision, what to study, who to hang out with, what to accept, what to let slide, what kind of person to allow yourself to be in an argument, you are back in that field, choosing a life package without properly seeing what comes inside. And, just as in the myth, after a while it becomes very hard to undo. The River of Forgetfulness is the perfect image of this: we forget that we are choosing. We act on autopilot, blame the circumstances, and only notice the package when it is already too late.

4. That is why philosophy matters.. and that is the real conclusion of the book

Here everything fits together. If the only thing separating a good choice from a disastrous one is the ability to examine the options before grabbing the first one that shines, then the most valuable skill there is consists in learning to examine. That is what Plato calls philosophy. Not memorizing Greek names: it is the training of looking at things calmly and seeing what they really are, not what they appear to be. It is exactly what Odysseus did while everyone else was trampling each other.

And so the Republic closes the circle. The initial question was “is it worth being just?”. The myth’s answer is: it is, because justice is not a rule imposed from outside that gets in your way, it is what keeps your soul in order, and a soul in order is the only thing capable of choosing well. Being unjust does not give you an advantage; it leaves you blind at the moment when seeing matters most.

“Keep thy heart with all diligence; for out of it are the issues of life.” — Proverbs 4:23

SOCRATES: Glaucon, you leave the palaestra like a man fleeing a fire. What happened?

GLAUCON: An argument, Socrates. The young men mocked old Ikkos, who teaches the wrestlers there. They said: “how can a man who never won at Olympia teach anyone to win?”. And I confess I laughed along with them, but now something bothers me.

SOCRATES: What bothers you?

GLAUCON: Ikkos trained three champions. None of his disciples ever asked about his crowns, and all of them left crowned. How can that be?

SOCRATES: Tell me: does the charioteer who wins the race win by his own eyes, or by the eyes of the horse?

GLAUCON: By his own, I suppose.

SOCRATES: And the man in the stands who notices that the charioteer takes the turn too early and loses half a length, does he himself need to know how to drive the chariot?

GLAUCON: Perhaps not. Seeing the error and performing the right move seem to be different things.

SOCRATES: Then you have already found two arts where you thought there was one. There is the art of doing, which lives in the hands and the legs, and there is the art of seeing and explaining, which lives in the eyes and in speech. Could the fastest runner in Athens tell you why he is fast?

GLAUCON: I doubt it. I once asked one of them and he only answered: “I have been running since I was a boy”.

SOCRATES: There is a man who knows and does not know that he knows. Put him to teaching and he will tell his pupils “run since you are boys”, and he will have taught nothing. And Ikkos, who may never have thrown an opponent, has watched ten thousand men fall and can name the cause of every fall. Which of the two will form the champion?

GLAUCON: Ikkos, it seems. Then it is settled: whoever teaches need not excel at what he teaches.

SOCRATES: Do not run so fast, Glaucon, or you will stumble like the charioteer at the turn. Answer me: would you agree to learn to swim from a man who has never entered the water?

GLAUCON: By no means!

SOCRATES: And why not, if a moment ago we agreed that teaching and doing are distinct arts?

GLAUCON: Because… that man would not know the water. He would speak of the sea as one speaks of a country never visited.

SOCRATES: Very well. Then distinguish with me: it is one thing to demand that the master be the best of practitioners; it is another to demand that he know from the inside what he teaches. Ikkos wrestled, fell, tasted the sand in his mouth. He did not gather crowns, but he gathered the knowledge of wrestling. What he lacks is glory; what he has is the science.

GLAUCON: But then why do we demand the crowns, Socrates? Why is the first question we ask a master “what have you won?”, and not “what do you understand?”.

SOCRATES: Because crowns can be counted on the fingers, and understanding cannot be counted. The hurried man prefers the proof the eyes verify in an instant to the proof that demands acquaintance and time. The crown is a label hung on the chest, and we have talked before, on another occasion, about how much labels spare us the work of thinking.

GLAUCON: Then are we wrong to demand practical reputation from those who teach?

SOCRATES: We are wrong and we are right, Glaucon, and that is what I ask you not to simplify. We are right, because practical reputation is a clue: whoever has done it probably knows it. We are wrong, because we take the clue for proof: whoever has done it may not know how to explain it, and whoever explains it may have known the thing without ever shining at it. The crown proves the man won; it does not prove he knows why he won, and even less that he knows how to make another win.

GLAUCON: And does the reverse also hold? Does the absence of a crown not prove ignorance?

SOCRATES: Just as the absence of wealth does not prove laziness, nor the absence of a beard prove youth. But note: it does not prove wisdom either. There are masters without crowns who are like Ikkos, and there are masters without crowns who are like the dry man teaching swimming. The absence of the label only gives back to you the work the label promised to spare: that of examining.

GLAUCON: And how do I examine, Socrates? If not by the master’s victories, then by what?

SOCRATES: By the fruits that are properly his own. The fruits of the wrestler are the falls he causes; the fruits of the master are the wrestlers he forms. Ask the master not “what have you won?”, but “who has learned from you, and what did they become?”. And ask yourself, as you listen to him: “do I understand more now than before?”. If the answer is yes, you have a master before you, whether he has a hundred crowns or none.

GLAUCON: Still, something in me resists. When the master also performs with excellence, is he not better?

SOCRATES: Perhaps he is rarer, Glaucon, as rare as a horse that is both beautiful and fast. When you find a man who does with excellence and explains with clarity, honor him, for nature has been generous twice. But do not make that double generosity the measure of all: you would demand of every horse that it be the most beautiful and the fastest, and you would be left on foot.

GLAUCON: So, in the end, is it right or wrong to demand of the one who teaches the reputation of the one who performs?

SOCRATES: That scale, Glaucon, is not mine to carry; it is carried by each person who chooses whom to trust with their learning. I only show you the weights: on one side, the honest clue that practice gives; on the other, the injustice of measuring one art with the ruler of another. Weigh them yourself, with every master you meet, and distrust whoever sells you the ready answer, including me, who never won a single bout and spend my days teaching in the square.

..then Glaucon fell silent, and Socrates smiled at his own example..

Glossary

• palaestra: a school of wrestling and physical exercise in ancient Greece
• charioteer: the driver of a horse-drawn racing chariot
• Olympia: the sanctuary where the ancient Olympic Games were held
• Ikkos: inspired by Ikkos of Tarentum, athlete and later a celebrated Greek trainer

Note

Does the one who teaches need to master, in practice, what they teach? The answer seems obvious, but maybe it is not. We tend to demand from those who teach the résumé of those who perform, as if the ability to explain were an inevitable consequence of experience. Yet there are those who never reached the greatest feats and still teach with clarity and depth. Likewise, there are extraordinary practitioners unable to transmit what they do.

Perhaps practical experience is a strong indication of competence, but not definitive proof of the ability to teach. And the absence of that experience is not enough, by itself, to invalidate a good master. Between practice and teaching there is a relationship, but not an identity. The question remains open: what, after all, qualifies someone to teach? Maybe the only certainty is the Socratic lesson that clues are not enough and that examining each case remains work without shortcuts.

No padding

Watch out for an increasingly common trap in the videos you watch: calculated stalling. It even has technical names. Padding is inflating a video with unnecessary content just to increase its length. Burying the lede is hiding the main information until the end, making you consume a lot of content before getting to what really matters. Watch time baiting is holding your attention for as long as possible to boost retention metrics. Pay attention to the signs: promises constantly postponed, unnecessary recaps and the feeling that the answer is always “just ahead”. When you notice this, remember: the video was not made to inform you, but to maximize the time you spend watching. Your time is valuable. Decide quickly whether it is worth continuing. If the content shows value, consume it in full. If you notice you are merely being retained, leave without guilt.

Clickbait

Be careful with titles made to exploit your curiosity and your fear of missing out. Phrases like “you need to know this”, “without this you will never make it”, “the mistake 99% of people make” or “nobody is telling you this” usually appeal more to emotion than to content. Good content clearly explains what you will learn. Clickbait makes vague, exaggerated or urgent promises to win your click before winning your trust. Before opening a video, ask yourself: does the title describe concrete information, or does it only stir anxiety and curiosity? If the promise is too generic, treat it with skepticism. Your click and your attention have value. Spend them only on content that delivers what it promises.

I think..

Be careful with content that presents personal opinions as if they were facts. Not every creator is transmitting knowledge; often they are sharing their own worldview, experiences and beliefs without making that explicit. Knowledge can be verified, questioned and compared against evidence. Opinions reflect individual interpretations and may vary from person to person. When that difference is not clear, it is easy to absorb a particular perspective as if it were a universal truth. Ask yourself: is this claim supported by data, evidence or reliable sources, or is it just the author’s conclusion? Developing this habit helps you learn more critically and build your own conclusions, instead of merely adopting someone else’s.

Perfect product (:

Remember that presentation is not content. Flawless production, animations, flashy slides, fast-paced editing and grandiose promises can make a message more convincing, but they do not make it truer or more useful. Evaluate first what is being said, not how it is being presented. Form may catch your attention; content is what should justify your time and your trust.

YouTube sophists

• use flashy titles and thumbnails to look like experts;
• speak with great confidence, even when the content is shallow;
• prioritize audience retention and engagement over depth;
• sell the feeling of learning, but deliver little practical knowledge.

A company’s market value is less an absolute truth and more a snapshot of the market’s perception at a given moment.

When someone decides to seriously study a new subject, they almost always experience a kind of intellectual anxiety. It is an understandable phenomenon. As the desire to know grows, so does the awareness of one’s own ignorance. The more you see, the more you notice what still remains hidden. “I know that I know nothing” — Socrates.

At that moment a problem appears that did not exist before: the limitation of time. It is no longer enough to want to learn; it becomes necessary to discern what deserves depth and what can remain in the realm of general information. Intellectual maturity consists precisely in this: knowing how far one needs to go and where it is wise to stop.

Looking at reality, one notices that there are, broadly speaking, three kinds of scholars. The first is the shallow generalist: knows a little about many things, but rarely reaches the essence of any of them. The second is the narrow specialist: masters a few subjects deeply, sacrificing breadth in favor of depth. The third combines both qualities: possesses vast culture and, at the same time, deep knowledge of the subjects studied. It is the rarest kind, because it demands not only intelligence, but discipline, time and an extraordinary capacity to order one’s own thinking.

The problem is that this third ideal, admirable as it is, becomes practically unattainable within the average lifespan of a human being, above all for those who have genuine curiosity about many areas of knowledge. An honest look at reality is enough to see that there is not enough time to absorb everything humanity has already produced, let alone keep up with what it keeps producing.

This is where the first great intellectual challenge appears: delimitation. Drawing a line is not an act of resignation, but of prudence. Without that delimitation, the student risks living permanently haunted by a sense of insufficiency, as if always behind in the face of the immensity of available knowledge. Clearly defined goals do not impoverish the intellectual search; on the contrary, they give it direction and meaning.

There is, however, a second problem just as important. Humanity has produced and keeps producing a colossal amount of content on practically every imaginable subject. Yet quantity has never been a synonym for quality. A large part of that production is born of naivety, lack of rigor, insufficient information or, worse still, the replacement of facts by opinions dressed up as knowledge. Not every author deserves trust. Not every work deserves attention.

That is why two virtues become indispensable to the student: the capacity to delimit one’s own field of investigation and the capacity to carefully select one’s sources. Before starting a journey of study, set aside some time to determine what you really need to learn and what level of mastery is necessary to reach your goal. When you reach it, consider the goal met. Otherwise, study turns into an endless search, incapable of producing any legitimate feeling of completion.

Likewise, set aside time to identify authors, teachers and thinkers who have withstood the judgment of time and whose reputation was built on intellectual seriousness. Prefer those who show prudence when evaluating their predecessors, who know the tradition they speak from and who have the honesty to clearly distinguish between what they know, what they believe and what they merely suppose. In matters of study, choosing the right guides is often more important than choosing the right subjects.

Everyone talks about the importance of having a solid base of knowledge, but few really understand what that means. When you study a subject and feel you are not understanding anything, or when you open a book and within the first pages find it excessively difficult or tedious, maybe the problem is neither the subject nor the book. Often, some earlier foundation is missing that would make the experience intelligible.

Try studying Differential and Integral Calculus without first mastering basic algebra. I lived that experience and I can say with confidence: it does not work. The intellect does not advance by arbitrary leaps. Knowledge has a structure, and each step presupposes the previous ones.

It took me a long time to understand the importance of logic, language, basic mathematics, epistemology, the scientific method, statistics and the history of ideas. For years I thought: “What am I ever going to need this for?”. And, in a certain sense, I was right. In the job market and in conventional everyday life, much of that knowledge seems dispensable. In most professions, it is perfectly possible to live without it.

But there is a difference between what is useful for earning a living and what is necessary for understanding life. For an existence guided by self-knowledge and the comprehension of reality, those studies are indispensable foundations. They are instruments of intellectual liberation.

It is worth asking yourself: does what I am studying have a relationship with who I am, or only with where I work?? Is it tied to my formation as a human being, or only to the next professional goal I want to reach?

That is a hard decision. What will you learn to develop as a person? And what will you learn to acquire practical skills, increase your economic usefulness and become more valued in the job market? In the end, both dimensions matter. I believe the mistake lies in completely sacrificing one of them in favor of the other.

I would like to close with a reflection..

How many people in your circle still carry the informal title of “healers”, keepers of knowledge passed down from generation to generation? How many will exist a hundred years from now? After answering in your head, leave the house, walk two blocks and look at the reality around you. You will most likely find a drugstore. They are everywhere: on corners, avenues, shopping centers and residential neighborhoods, a few minutes away for most of the population.

This is not about rejecting the advances of medicine nor idealizing the past. The question is another: we are becoming more and more dependent on systems we understand less and less. We have lost touch with traditions, local knowledge and forms of understanding that for centuries were part of human experience.

I have the impression that we are forming individuals ever more specialized in operating mechanisms and less capable of understanding themselves. Knowledge grows in volume, but wisdom seems to become rarer.

I know.. this text started out talking about knowledge and ended with my indignation :/ . Maybe because, deep down, knowledge is not just accumulating information, but understanding the place we occupy in the world and the direction we are walking in.

Finally, I remember a sentence I once heard from a cardiologist who has since passed away:

“Only knowledge frees man.” — E.C.

Knowledge vs. Intelligence vs. Wisdom

Knowledge is the set of information, concepts and facts a person has acquired through study, observation or experience. It is what someone knows. A doctor knows anatomy, physiology and pharmacology because they studied those disciplines; a mechanic knows how engines work because they learned their structure and operation. Knowledge mainly answers the question: “what is it?”.

Intelligence is the capacity to understand, relate, analyze and use knowledge to solve problems or deal with new situations. Two people may have the same amount of knowledge, yet show different levels of intelligence when applying it. A student who learns the formulas of physics has knowledge; one who can use them to solve a problem never seen before shows intelligence. Intelligence answers the question: “how do I use what I know?”.

Wisdom is the capacity to correctly judge the value, the limits and the consequences of knowledge and intelligence in real life. It is a virtue tied to prudence, discernment and the understanding of human nature. An investor may have knowledge of economics and the intelligence to identify financial opportunities, but only wisdom will let them notice when they already have enough, when the risk is excessive or when the pursuit of money is sacrificing more important aspects of life. Wisdom answers the question: “is it worth doing, and what will the consequences be?”.

In simple terms, knowledge is knowing that fire burns; intelligence is understanding how to produce, control or use fire; wisdom is knowing when to light it, when to put it out and when to keep away from it. Knowledge accumulates information, intelligence turns information into solutions, and wisdom steers those solutions toward truly good and fitting ends.

This distinction appears, in different formulations, in authors such as Aristotle, who differentiated science, craft and prudence, and also in modern thinkers of epistemology and cognitive psychology. In every serious tradition of thought, the conclusion is similar: it is possible to have a lot of knowledge without being intelligent, to be intelligent without being wise, but it is impossible to be truly wise without some degree of knowledge and intelligence. Wisdom does not replace the two; it orders them.

Being a polymath

A polymath is someone whose mind transcends the artificial borders between fields of knowledge, identifying the principles, patterns and structures that connect distinct areas into a unified understanding of reality. Rather than merely accumulating information, the polymath synthesizes, reconstructs and integrates knowledge, turning learning into systemic perception. Their cognition does not operate linearly or in compartments, but as a dynamic network of connections, in which science, philosophy, psychology, politics, finance and human experience reveal different expressions of the same fundamental truths.

SOCRATES: Glaucon, I see you frowning. What troubles you?

GLAUCON: I have discovered, Socrates, that a poet whose work I dearly love holds political opinions I find repugnant. And another, whose ideas I share as a brother would, writes dreadful verses. I feel I must choose one and renounce the other.

SOCRATES: Tell me: when you eat from a fig tree, do you judge the fruit by its taste, or by the opinions of the farmer?

GLAUCON: By its taste.

SOCRATES: And if the farmer is a wicked man, but his figs are sweet, will you stop eating them?

GLAUCON: No, for the fig has nothing to do with the man’s wickedness.

SOCRATES: Then does the work, once made, live on its own, or does it depend at every instant on whoever made it?

GLAUCON: It lives on its own.

SOCRATES: So we may judge the fig by what it is, without being obliged to praise the farmer in everything. And conversely: the fact that a man is virtuous will not make a bitter fig sweet. Mix these things up, Glaucon, and you lose twice: you lose the taste of good figs, and you lose the clarity to judge the man.

GLAUCON: I understand. But there is yet another thing on this tablet that bothers me. It is written about me: “Glaucon is a fool.”

SOCRATES: Just that?

GLAUCON: Just that.

SOCRATES: And if I said to you: “Glaucon, you are a fool because, calculating the price of three amphorae of oil, you multiplied by two and lost a drachma,” what would you do?

GLAUCON: I would check whether it is true, and, if it were, I would correct the error.

SOCRATES: And if I merely said “you are a fool,” and nothing more?

GLAUCON: I would not know what to correct, nor whether there is anything to correct.

SOCRATES: So one statement hands you a reason, a path. The other hands you nothing but a thrown stone. Tell me: would you entrust your horse to a man who holds out his hand for the reins without saying who he is?

GLAUCON: Never.

SOCRATES: And your judgment of yourself, a thing far more valuable than a horse, will you hand it to whoever labels you without saying why?

GLAUCON: I would be more of a fool than he accuses me of being.

SOCRATES: Then keep this rule: the accusation without a reason is chaff in the wind, ignore it. The accusation with a reason is a seed that may sprout in you, examine it. But mark this well, Glaucon: no one can examine for you. Not I, not the author of the tablet, not the crowd in the square.

GLAUCON: How so, Socrates?

SOCRATES: I mean that the work of weighing reasons, accepting what holds and discarding what does not, is yours and yours alone. And this is not only true of men, Glaucon. It is true of everything you encounter: a law imposed on you, advice you receive, a decision you must make in your house or in your city. In all of them, someone will want to think for you.

GLAUCON: And if I err by deciding alone?

SOCRATES: You will err. And that is what distinguishes the free man from the slave: the free man carries the consequences of what he concludes. If you decide well, you will reap. If you decide badly, you will lose. But in either case you will have learned something no one can learn for you.

GLAUCON: And the one who prefers that another decide in his place?

SOCRATES: He lives a borrowed life. When something goes right, the merit will not be his. When something goes wrong, he will blame the other, and learn nothing, for the error was not properly his own. It is the worst of conditions, Glaucon: that of the man who never fails because he never dared to judge.

GLAUCON: Then freedom comes at the price of responsibility.

SOCRATES: That is its price, and also its dignity. Examine every work, every accusation, every situation. Always ask for the reasons. Decide. And when the consequence comes, sweet or bitter, receive it as your own. It is the only road to truly becoming someone.

..then Socrates straightened his cloak, Glaucon turned the tablet to face the wall, and the two went for a beer.

Glossary

• amphorae: ancient vessels used to store liquids such as wine and oil
• drachma: a coin used in ancient Greece
• portico: a columned entrance at the front of a building

Note

Reading the Greek texts means not treating them as mere discursive constructions, but as records of an effective perception of reality, guided by the attempt to grasp the essence and inner workings of the objects of nature.

When I read Plato’s Republic, I knew I did not have, and perhaps still do not have, enough grounding to make the most of it. Even so, I felt it was a reading worth doing. I want to reread it soon, precisely to see how the experience of a work changes as we mature as readers.

What motivated me most was the desire for a holistic view, of both the product and the company. I liked understanding the parts and seeing how they connect and interact. That happened naturally and, in the same way, I developed the urge to express the health of systems, their dependencies and their integrations through dashboards, monitoring them and following their evolution. That led me to a second move, into platform engineering, specifically observability platforms, where I spent my last 12 months.

What you will see on this blog is a mix of what I lived through in that time: mostly lessons that cost me dearly and tips about technical work inside large organizations, which often have rigid processes and an anti-productive culture in the name of stability. But you will also see many philosophical texts, inspired by the works I have been reading and by the thoughts and reflections I have been having. That is part of me, and I would like it to live in the same place.

Along the way, if you read my texts, you will end up getting to know me better.

Every post is available in Portuguese and English; use the PT | EN switch at the top of the page. Besides the posts, there are two more things around here:

• Thoughts: quotes from others and a few thoughts of my own, with dates.
• Cheatsheet: day-to-day commands and tricks, with search and tag filter.

Find me on GitHub and X.

“The most important fact that sets the human being apart from all its brothers and cousins on the phylogenetic scale is knowledge; only knowledge frees man, only through knowledge is man free and, being free, he can aspire to a better condition of life for himself and all his fellow beings. I can only understand a society in which knowledge is the foremost reason the government gives for the formation of the citizen. My message is a positive one: that man must know, must understand, and in understanding, he is free.”

— Dr. Enéas Ferreira Carneiro (Rio Branco, November 5, 1938 — Rio de Janeiro, May 6, 2007)