I have a page for every month. Every day, I write out what I did. At the end of the month, I make a drawing in the header to show what I did that month. Here are a few of the drawings:

In January, I started a Substack, made paintings for friends, and wrote up two Substack posts on security.

In February, I learned took a CSS course and created a component library for myself.

In March, I read a few books, worked on a writing app, took a trip to New York, and drafted several posts on linear algebra for this Substack.

(If you’re wondering where these posts are, there’s a lag time between draft and publish, where I send the posts out for technical review and do a couple of rounds of rewrites).

Pros

I don’t really spend much time celebrating my accomplishments. Once I accomplish something, I have a small hit of, “Yay, I did it,” before moving on to, “So, what else am I going to do?” For example, when I finished my book (a three-year-long effort), I had a couple of weeks of, “Yay, I wrote a book,” before this became part of my normal life, and it turned into, “Yes, I wrote a book, but what else have I done since then?”

I thought the tada list would help reinforce “I did something!” but it also turned into “I was able to do this thing, because I did this other thing earlier”. I’ll explain with an

example

For years I have been wanting to create a set of cards with paintings of Minnesota, for family and friends. The problem: I didn’t have many paintings of Minnesota, and didn’t like the ones I had.

So I spent 2024 learning a lot about watercolor pigments, and color mixing hundreds of greens, to figure out which greens I wanted to use in my landscapes:

Then I spent the early part of 2025 doing a bunch of value studies, because my watercolors always looked faded:

(Value studies are where you try to make your paintings look good using black and white only, so you're forced to work using value instead of color. It’s an old exercise to improve your art).

Then in the summer, I did about 50 plein air paintings of Minnesota landscapes:

(Plein air = painting on location. Please admire the wide variety of greens I mixed for these paintings).

Look at how much better these are:

Out of those 50, I picked my top four and had cards made. Thanks to the “tada” list, it wasn’t just “I made some cards”, it was

• Remember when I spent countless hours on color mixing

• And value studies

• And spent most of my summer painting outside?

The payoff for all that work was these lovely cards.

Another example

For a while now, I have wanted a mustache-like templating language, but with static typing. Last year, I created a parser combinator library called `tarsec` for TypeScript, and this year, I used it to write a mustache-like template language called `typestache` for myself that had static typing.

I’ve since used both `tarsec` and `typestache` in personal projects, like this one that adds file-based routing to express and autogenerates a client for the frontend.

Part of the reason I like learning stuff is it lets me do things I couldn’t do before. I think acknowledging that you CAN do something new is an important part of the learning process, but I usually skip it. The tada list helps.

Cons

Maybe the most obvious con: a tada list forces you to have an accomplishment each day so you can write it down, and this added stress to my day.

Also, a year is a long time to keep it going, and I ran out of steam by the end. You can see that my handwriting gets worse as time goes on

and for the last couple of months, I stopped doing the pictures.

In conclusion

It’s fun to see things on the list that I had forgotten about. For example, I had started this massive watercolor painting of the Holiday Inn in Pacifica in February, and I completely forgot about it

Will I do this next year? Maybe. I need to weigh the accomplishment part against the work it takes to keep it going. It’s neat to have this artifact to look back on either way.

Postscript

A few more of the several color studies I did:

Including another grid of greens.

Picking a city

When my wife and I were deciding which city to live in, we made a list of cities, and scored each city based on some criteria. Here’s San Francisco and Minneapolis, for example, on weather and affordability.

You can see we loved the weather in San Francisco, but Minneapolis was way cheaper to live in. After this was done, we just added up the columns to figure out which city to live in!

Here’s the thing, though: I really liked the weather in San Francisco. I wanted some way to do this calculation, but have the weather matter more. Well, I could do that by using weights.

If I wanted the weather to matter 10% more, I could multiply by 1.1 before doing the addition.

(I also multiplied the affordability by 1 to show that I’m keeping it the same).

This is the essence of what a dot product is! Earlier I was adding up the numbers. Now I’m weighting the numbers before I add them

A dot product is a type of weighted sum.

Dot product using vectors

Remember vectors from last chapter? Well, the dot product is an operation you perform on two vectors. Let’s write the above as a vector. For example, here are the scores for San Francisco as a vector

Here are the weights as a vector

We simply multiply the numbers by the weights and then add:

Tada! We just took a dot product of two vectors! It’s a straightforward operation.

Three cities

Let’s see the same example with three cities. Here are our three cities with scores for weather and affordability:

The simple way to calculate scores would be to just add up the numbers:

But instead, we’re going to take the dot product:

We are taking three separate dot products here. For each city, we multiply its scores by the weights:

I’m saying that now to make it clear that we’re not taking the dot product of three vectors. That’s impossible, we can only take the dot product of two vectors. Instead we are taking three separate dot products. More on this later.

Now let’s look at another example. Let’s look at the Minnesota lottery.

The Minnesota Lottery

It takes $2 to buy a ticket for the Minnesota lottery. Here are the odds:

So your odds of winning $2 are 1 in 17, your odds of winning $20 are 1 in 2404, etc. Given this information, how do you calculate how much a single ticket is worth, on average?

To find out, we again need to take the dot product. Here are the two vectors:

Prize money on the left, probability of winning on the right. Let’s see the calculation:

The ticket is worth $1.17176. It costs $2, so on average you can expect to lose money, which is what we knew already.

Same as the cities example, we are weighting the numbers, except this time the weights are the probability that we win that much money. The final number is called the expected value.

It’s the expected value of our ticket.

That’s all for dot products. It’s a straightforward operation, but one that’s important to know for matrix multiplication, which is the topic of the next chapter.

Summary

A dot product is a type of weighted sum, and is an operation you do on two vectors. You multiply each element of the vectors together, then add up the results:

There are two big ideas I want to introduce in the first chapter: Gaussian elimination, (which is not strictly a linear algebra thing, and has been around for years before linear algebra came along), and row picture versus column picture, which is a linear algebra thing.

Money example

Let’s say you have a bunch of nickels and pennies, and you want to know how many of each do you need to have 23 cents.

You could write that as an equation that looks like this:

x is the number of nickels you need, y is the number of pennies you need. And you need to figure out the x and y values that would make the left-hand side work out to 23. And this one is pretty easy, you can just work it out yourself. You’d need four nickels and three pennies.

So x is four, y is three.

This kind of equation is called a linear equation. And that’s because when you plot this equation, everything is flat and smooth. There are no curves or holes. There isn’t a ^2 in the equation for example to make it curved. Linear equations are great because they’re much easier to work with than curved equations.

Aside: Another solution for the above is 23 pennies. Or -4 nickels + 43 pennies.

The point is you have two variables (x and y for nickels and pennies), and you are trying to combine them in different ways to hit one number. The trouble starts when you have two variables, and you need to combine them in different ways to hit two different numbers. That’s when Gaussian elimination comes in. In what world would you have to hit two different numbers? Does that seem outlandish? It’s actually very common! Read on for an example.

Food example

Now let’s look at a different example. In the last one we were trying to make 23 cents with nickels and pennies. Here we have two foods. One is milk, the other is bread. They both have some macros in terms of carbs and protein:

and now we want to figure out how many of each we need to eat to hit this target of 5 carbs and 7 protein.

This is a very similar question to the one we just asked with nickels and pennies, except instead of one equation, we have two equations:

Again we have an x and a y. Lets find their values. To solve these kinds of questions, we usually use Gaussian elimination. If you’ve never used Gaussian elimination, strap in.

Gaussian elimination

Step one is to rewrite this as a set of two equations:

Now you subtract multiples of one equation from another to try to narrow down the value of one variable. Lets double that second equation:

See how we have a +2y and a -2y now? Now we can add the two equations together to eliminate y:

We’re left with one equation and one variable. We can solve for x:

Aha, we know x = 3. Now we can plug that into one of the equations to find y.

We plug that in to one of the equations and find out that y equals 1, and there we have answer: three milks, one bread, is what we need.

This method is called Gaussian elimination, even though it was not discovered by Gauss. If you haven’t seen Gaussian elimination, congratulations, you learned a big idea! Gaussian elimination is something we will talk about more. It’s part of what makes linear algebra useful.

We can also find the solution by drawing pictures. Let’s see how that works.

Picture version

Let’s plot one of these lines. First, we need to rewrite the equations in terms of x.

Reminder: first equation is for carbs, second for protein. x is number of milks, y is number of breads.

Now let’s plot the graph for the first equation.

Now, what does this line represent?

It’s all the combinations of bread and milk that you can have to get exactly five carbs:

So you can eat no milk and two-and-a-half breads, or two milks and one-and-a-half breads, or five milks and no bread, to get to exactly five carbs. All of those combinations would mean you have eaten exactly five carbs. You can pick any point that sits on this line to get to your goal of eating five carbs.

Note: You can see the line goes into the negative as well. Technically, 5 breads and -5 milks will give you 5 carbs as well, but you can’t drink negative milks. For these examples, let’s assume only positive numbers for the variables.

Now, let’s plot the other one. This is the same thing, but for protein.

If you eat any of these combinations, you’ll have met the protein goal:

You can pick a point that sits on the first line to meet the carb goal. You can pick a point that sits on the second line to meet the protein goal. But you need a point that sits on both lines to hit both goals.

How would a point sit on both lines? Well, it would be where the lines cross. Since these are straight lines, the lines cross only once, which makes sense because there’s only a single milk and bread combo that would get you to exactly five grams of carbs and seven grams of protein.

Now we plot the lines together, see where they intersect, and that’s our answer:

Bam! We just found the solution using pictures.

So that’s a quick intro to Gaussian elimination. But you don’t need linear algebra to do Gaussian elimination. This is a technique that has been around for 2,000 years. It was discovered in Asia, it was rediscovered in Europe, I think in the 1600s or something, and no one was really talking about “linear algebra”. This trick is just very useful.

That’s the first big idea you learned. You can stop there if you want. You can practice doing this sort of elimination. It’s a very common and useful thing.

The column picture

What we just saw is called the “row picture”. Now I want to show you the column picture. I’m going to introduce a new idea, which is: instead of writing this series of equations, what if we write just one equation? Remember how we had one equation for the nickels and pennies question?

What if we write one like that for food? Not a system of equations, just a single equation? What do you think that would look like? Something like this:

It’s an equation where the coefficients aren’t numbers, they’re an “array” of numbers. The big idea here is: what if we have a linear equation, but instead of numbers, we have arrays of numbers? What if we treat [1 2], the way we treat a number?

Can that actually work? If so, it is pretty revolutionary. Our whole lives we have been looking at just numbers, and now we’re saying, what if we look at arrays of numbers instead?

Let’s see how it could work in our food example. What if the coefficients are an array of numbers? Well, this way of thinking is actually kind of intuitive. You might find it even more intuitive than the system of equations version.

Each of these coefficients are called vectors. If you’re coming from computer science, you can kind of think of a vector as an array of numbers (i.e. the order matters).

Lets see how we can use vectors to find a solution to the bread and milk question.

Step one: graph the vectors.

Yeah, we can graph vectors. We can graph them either as a point, like I’ve done for the target vector here, or as an arrow, which is what I’ve done with the vector for bread and the vector for milk:

Use the two numbers in the vector as the x and y coordinates.

That is another big idea here: We always think of a set of coordinates giving a point, but you can think of vectors as an arrow instead of just a point.

Now what we’re asking is how much milk and how much bread do we need, to get to that point?

This is a pretty simple question. It’s simple enough that we can actually see it. Let me add some milks:

And let me add a bread. Bingo bango, we’re at the point:

Yeah, we literally add them on, visually. I personally find this more intuitive. I think the system of equations picture can confuse me sometimes, because the initial question was, “how much bread and how much milk should I eat?” The vector way, you see it in terms of breads and milks. The row way, you see it as one of the lines is the carbs, the other line is the protein, and the x and y axes are the amount of bread, which results in the same thing, but it’s a little more roundabout, a little more abstract. This one is very direct.

The algebra way

We just saw that we can graph vectors too. Graphing it works differently from graphing the rows, but there is a graph we can make, and it works, which is pretty cool. What about the algebra way?

Here is the equation again:

Since we already know the answer, I’ll just plug that in:

Now, the question is how does the left side equal the right side? The first question is how do you define this multiplication? Well, in linear algebra, it’s defined as, if you multiply a scalar by a vector, you just multiply it by each number in that vector:

Now you are left with two vectors. How do you add two vectors? Well, in the linear algebra you just add the individual elements of each vector:

And you end up with the answer.

Congratulations, you’ve just had your first taste of linear algebra. It’s a pretty big step, right? Instead of numbers, we’re working with arrays of numbers. In future chapters, we will see why this is so powerful.

That’s the first big concept of linear algebra: row picture vs column picture.

Finally, I’ll just leave you with this last teaser, which is: how would you write these two equations in matrix notation? Like this:

This is the exact same thing as before. You can write it as scalars times columns, as we had done before:

or you can write it as a matrix times a vector, as above. Either one works.

Matrices are a big part of linear algebra. But before we talk about matrices, we will talk about the dot product, which is coming up next.

More chapters coming soon! Subscribe to get an email when they come out.

Additional reading

• Check out Gilbert Strang’s lectures on linear algebra on YouTube.

Summary

P.S. Want more art? Check out my Instagram.

OAuth was first introduced in 2007. It was created at Twitter because Twitter wanted a way to allow third-party apps to post tweets on users' behalf. Take a second to imagine designing something like that today. How would you do it? One way would just be to ask the user for their username and password. So you create an unofficial Twitter client, and present the user a login screen that says "log in with Twitter". The user does so, but instead of logging into Twitter, they're actually sending their data to you, this third-party service which logs into Twitter for them.

This is bad for a lot of reasons. Even if you trust a third-party app, what if they don't store your password correctly and someone steals it? You should never give your password to a third-party website like this.

Another way you might be thinking is, what about API keys? Because you're hitting Twitter's API to post data for a user, and for an API, you use API keys. But API keys are general. What you need is an API key specific to a user.

To solve these problems, OAuth was created. You'll see how it solves all these problems, but the crux of OAuth is an access token, which is sort of like an API key for a specific user. An app gets an access token, and then they can use that to take actions on the user's behalf, or access data for a user.

How OAuth works

OAuth can be used in a lot of different ways, one of the reasons it is so hard to understand. In this post, we’re going to look at a typical OAuth flow.

The example I'm going to use is YNAB. If you haven't used it, YNAB is like a paid version of Mint. You connect it to a bank account, and then it pulls all your transactions from that account, and shows them to you with very pretty charts. You can categorize your spending, and then it tells you for example, hey, you're spending too much on groceries. It helps you manage your finances. So, I want to use YNAB, and I want to connect it to Chase Bank, but I don't want to give it my Chase password. So instead, I'm going to use OAuth.

Let's look at the flow first, and then let's understand what's going on. We're actually going to look at the flow twice, because I think you need to look through an OAuth flow at least two times to understand what's going on.

OAuth flow, take 1

So to start, I'm at YNAB, and I want to connect Chase as a source. The OAuth flow looks like this:

1. YNAB redirects me to Chase.

2. At Chase, I log in with my username and password.

3. Chase shows me a screen saying "YNAB wants to connect to Chase. Pick what accounts you want to give YNAB access to". It'll show me a list of all my accounts. Let's say I pick just my checking account, to give YNAB read access to this account, and hit OK.

4. From Chase, I'm redirected back to YNAB, and now, magically, YNAB is connected to Chase.

This is the experience from a user's perspective. But what happened there? What magic happened in the background, so that YNAB somehow has access to my data on Chase?

The end goal is to end up with an access token

Remember, the end goal of OAuth is for YNAB to end up with an access token, so it can access my data from Chase. Somehow, as I went through this flow, YNAB ended up with an access token. I'll spoil the surprise by telling you how it got the access token, and then I'll walk you through what happened in more detail.

A quick word on security

How does Chase give YNAB the access token? When you were redirected from Chase back to YNAB, Chase could have just added the access token in the URL. It could have redirected you back to a URL like this:

https://www.ynab.com/redirect?access_token=123

and then YNAB would be able to get the access token.

BAD IDEA!!

An access token is supposed to be secret, but URLs can end up in your browser's history or some server logs, in which case it's easy for anyone to see your access token.

So Chase could technically redirect you back to YNAB with the access token in the URL, and then YNAB would have the access token. End of OAuth flow. But we don’t do it this way, because sending an access token in the URL is not secure.

When you were redirected from Chase back to YNAB, Chase sent YNAB an authorization code in the URL.

An authorization code is not an access token! Chase sends YNAB an authorization code, and YNAB exchanges the authorization code for an access token. It does this by making a backend request to Chase, a backend POST request over HTTPS, which means no one can see the access token.

And then YNAB has the access token. End of OAuth flow. OAuth success.

Two parts of OAuth

Let's talk about what we just saw. At a high level, there are two parts to an OAuth flow. The first is the user consent flow, which is where you, the user, log in and pick what to give access to. This is a critical part of OAuth, because in OAuth, we always want the user to be actively involved and in control.

The other part is the authorization code flow. This is the flow where YNAB actually gets this access token.

Let's talk about more details of exactly how this works. And let's also talk about some terminology, because OAuth has very specific terminology.

• Instead of user, we say resource owner.

• Instead of app, we say OAuth client or OAuth app.

• The server where you log in is called the authorization server. The server where you get user data from is called the resource server (This could be the same as the authorization server).

• On the authorization server, when the user picks what's allowed, those are called scopes.

I'll try to use that terminology, because you'll need to get familiar with it if you're going to read more OAuth documentation.

So let’s look at this high level again, with the new terms.

OAuth flow, take 2

You have OAuth clients. An OAuth client wants to access data on a resource server, and the data belongs to the resource owner.

To do that, the OAuth client redirects to the authorization server. The user logs in, user agrees to scopes (what this token is allowed to access), and the user gets redirected back to the OAuth client with an authorization code in the URL.

On the back end, the OAuth client sends the authorization code and client secret (we'll talk about client secrets shortly) to the authorization server, and the authorization server responds with the access token.

That's the exact same flow, but using the new terminology we just discussed. Now let's talk specifics. We've seen what this flow looks like from the user's point of view, let's look at what it looks like from the developer's point of view.

Registering a new app

To use OAuth, you first need to register a new app. So for example, GitHub provides OAuth. If you want to create a new app for GitHub, you first register it. Different services require different types of data in the app registration, but every service will require at least

1. an app name, because when the user goes to GitHub, for example, GitHub needs to be able to say "Amazon Web Services is requesting read access to your repos and gists"

2. A redirect URI. And we'll talk about what that is shortly.

GitHub will respond with:

1. A client ID. This is a public ID that you'll be using to make requests

2. A client secret. You'll be using this to authenticate your request.

Awesome, you have registered your OAuth application. Let's say your app is YNAB, and one of your users wants to connect to Chase. So you start a new OAuth flow... your very first one!

Step one: You will redirect them to Chase's authorization server's OAuth endpoint, passing these parameters in the URL:

1. Client ID, which we just talked about.

2. The redirect URI. Once the user is done on Chase, this is where Chase will redirect them back to. This will be a YNAB url, since you're the YNAB app.

3. Response type, which is usually "code", because we usually want to get back an authorization code, not an access token, which is less secure.

4. Scopes. So what scopes are we requesting? i.e. what user data do we want to access?

This is enough information for the authorization server to validate the request and show the user a message like "YNAB is requesting read access to your accounts".

How does the authorization server validate the request? Well, if the client ID isn't valid, the request is invalid right away. If the client ID is valid, the authorization server needs to check the redirect URI. Basically, since the client ID is public, anyone could go get the YNAB client ID, and create their own OAuth flow that hits Chase, but then returns the user back to, let's say, evildude.com. But that's why when you register your app, you have to tell Chase what a valid redirect URI looks like. At that point, you would tell Chase that only YNAB.com URIs are valid, thus preventing this evildude.com scenario.

If everything is valid, the authorization server can use the client ID to get the app name, maybe the app icon, and then show a user consent screen.

The user will click which accounts they want to give YNAB access to, and hit okay.

Chase will redirect them back to the redirect URI that you gave, lets say ynab.com/oauth-callback?authorization_code=xyz.

Side note: you might be wondering, what is the difference between URI and URL? Because I'm kind of using both. Well, a URL is any website URL that we know and love. URI is more general. URL is a type of URI, but there are many other types of URIs.

The reason I'm saying redirect URI instead of redirect URL is because mobile apps won't have a URL. They'll just have a URI, which is a protocol they have made up that might look something like myapp://foobar. So if you're only doing web work, whenever you read URI, you can read it as URL. And if you're doing mobile work, you can read URI and know that yes, your use case is supported too.

So user is redirected back to ynab.com/oauth-callback?authorization_code=xyz, and now your app has an authorization code. You send that authorization code to the Chase authorization server, along with your client secret. Why include the client secret? Because again, the authorization code is in the URL. So anyone can see it and anyone could try to exchange it for the access token. That's why we need to send the client secret, so Chase's server can say "Oh yes I remember I had generated this code for this client ID, and the client secret matches. This is a valid request."

And then it returns the access token. Note how in every step of the OAuth flow, they have thought through how someone could exploit the flow, and added safeguards*. That is a big reason why it's so complicated.

*I'm reliably informed by a friend in security that the OAuth designers learned a bunch of lessons the hard way, and that is another reason why it is so complicated: because it had to be patched repeatedly.

The other big reason is because we want the user to be involved. That makes it complicated because all the user stuff has to be frontend, which is insecure, because anyone can see it. And then all the secure stuff has to be on the back end.

I keep saying frontend and back-end, but in the OAuth docs, they say front-channel and back-channel instead. Let's talk about why.

Front-channel and back-channel

So, OAuth doesn't use the terms frontend and back-end, it uses front-channel and back-channel. Front-channel means GET requests, where anyone can see the parameter in the URL, and back-channel means POST requests, where that data is encrypted (as part of the POST body). The reason OAuth doesn't use frontend or backend is, because you could make POST requests using JavaScript! So, theoretically, you could exchange your authorization code for an access token right on the frontend, in JavaScript, by making a POST fetch request.

Now, there is a big problem with this, which is you also need the client secret to make that request. And of course, once the secret is on the frontend and accessible in JavaScript, it's not secret anymore. Anyone can access it. So, instead of using the client secret, there's a different way to do it called PKCE, spelled P-K-C-E, pronounced “pixie” (seriously). It's not as secure as doing it on the backend with the client secret, but if backend is not an option for you, you can do it using PKCE. So just know that if you have an app without a back-end, you can still do OAuth.

I may cover PKCE in a future post, as it is now recommended for the standard flow as well, since it helps protect against auth code interception.

Same problem for mobile apps. Unless you have a mobile app that has a backend component, like a backend server somewhere, if you're putting your client secret in a mobile app, well, anyone can get that because there are tons of tools to extract strings from mobile apps. So, instead of including your client secret in your app, you should again use PKCE to get that access token.

So those are two other terms that are good to know: front-channel and back-channel.

At this point, you've seen what the OAuth flow looks like from the user's perspective, and from the developer's perspective, and you have seen the components that make it secure.

Some final thoughts

One last thing I want to mention is OAuth can look like a lot of different ways. I covered the main recommended OAuth flow above, but some people may do OAuth by passing back an access token in the redirect instead of the authorization token (doing that is called the "implicit flow"). Some people may do it using PKCE. There's even a way to do OAuth without the user consent part, but that really is not recommended.

The other part of OAuth we didn't cover is that tokens expire and you need to refresh them. And that happens through a refresh flow. Also, OAuth is all about authorization, but some workflows use OAuth to log in, such as when you use a “sign-on with Google” feature. This uses OpenID Connect, or OIDC, which is a layer on top of OAuth that also returns user data instead of just an access token. I'm mentioning this here because when you look for OAuth on the web, you'll see a lot of different flows, and you may be confused as to why they're all different. And the reason is, OAuth is not straightforward like HTTP, OAuth can look a lot of different ways.

Now you're good to go out and do your own OAuthing. Good luck!

Story time

Every Wednesday, I work with a local Minnesota artist who helps me improve my watercolor skills. One day, he was giving me some advice on my painting, and he said: "The sky needs to be a warm blue, so use ultramarine." This surprised me because I've always thought of ultramarine as a cool blue.

So I conducted a poll on Instagram. I posted the same three images as above, and asked people to identify the warmer blue, just like I asked you.

The results surprised me. They were almost evenly 50/50!

Results

Lets look at the results:

Personally, I’d pick B, A, A. I was surprised at how split the crowd was. A couple of art friends messaged me, also surprised, and questioning whether they knew anything about art at all. Because as artists, it is so obvious what the warm blue is! Except as it turns out, it isn’t.

Here’s the explanation: in each case, the blue on the right is the redder blue.

Controversy!!

I did a little Googling and found out this topic is hotly contested. Here's a thread specifically on ultramarine that goes on for 11 pages!

Lots of people saying they think Ultramarine is cool:

I hear artists refer to Ultramarine as a warm blue and Phthalo as a cool blue. To me eyes, when I look at the two colors it seems just the opposite to me

I believe Ultramarine is cool and Pthalo warm.

This thread tries to bring some science into it:

Those closer to yellow-orange/red-orange will be “warmer”, while those closer to blue-green will be “cooler”. This is why Ultramarine Blue (closer to red-orange) is warmer compared to Phathlo Blue (which is a blue-green).

This is similar to what a few art friends told me: Red is a warm color, and green is a cool color, so blue + red = warmer, blue + green = cooler.

Issue solved.

But wait!! Here comes Reddit with a different opinion!

It is a warm blue because it has yellow in it.

Ah, so blue + purple = cooler, blue + yellow = warmer!

blue + red = warmer
blue + yellow = warmer

Both make sense!! Which is right??

Well, you're going to hate this answer. Because not only is there not a right answer, the color wheel is wrong.

The outdated color wheel

Everyone knows the primaries, right? Red, yellow, and blue are primary colors, and they mix to form the secondary colors. Everyone knows that.

But if that's the case, why are printer cartridges filled with CMYK (Cyan, magenta, yellow, black)? Shouldn't they be RBYK?

Here's the thing: the color wheel was created a long time ago, when artists first figured out they could mix colors to create new colors. At that time, they had limited pigments, and red, yellow, and blue were the best primary colors to use.

We have many more pigments now. Red, yellow, and blue are no longer the best primaries: cyan, magenta, and yellow are. But we still keep teaching kids the wrong color wheel.

And another thing: why do we have the same number of steps between yellow and blue as between yellow and red? I would say yellow to blue is a big shift (warm to cool), whereas yellow and red are much more similar.

Now at this point, some of you are probably saying to yourselves, "I am so happy to be reading this. It's time that someone exposed the arrogance of the color wheel community."

Others of you are probably saying to yourselves, "Wow, I've never read a blog post and been left with less information than I had before. When I started reading this, I didn't care about warm blues and I understood the color wheel. Now I don't understand either."

For all of you in the second group, here is a possible solution to one problem.

The Munsell color wheel

Some artists have been using the Munsell color wheel instead. This color wheel gives a lot more steps between yellow and blue. We actually have a lot of nuanced blue-green colors, like emerald, teal, etc. and it's useful to have a color wheel that gives space for those.

Maybe these extra steps will help being nuance to the warm blues discussion. Or maybe the reality is you cannot analyze color by itself, but only in relation to colors that is near, as some artists think. Maybe all blues can be warm or cool, depending on situation.

Final thoughts

We talked about color temperature. We talked about color wheels. Finally, here is a cool icebreaker for your next meeting: Show people the three images above, and ask them to pick the warmer blue. I guarantee you this will lead to an interesting discussion.

Other interesting things:

• Colour story: Ultramarine

• My instagram

• The artist who’s teaching me watercolor

• Color fidget: a fidget that uses the Munsell wheel

But one last thing to mention first:

Public vs Elastic IPs

One thing we haven't talked about is public IP addresses. All the IP addresses I've mentioned so far are local to within the VPC, but you also need some sort of public IP address that people on the internet can visit to connect to your server.

You can create an EC2 instance with a public IP address. The key thing to note with public IPs though, is whenever you stop your instance, the public IP will change, so it’s not a good idea to point your domain name to your public IP. For that you need an Elastic IP. Request an Elastic IP from AWS, and then you can attach it to a particular EC2 instance (or to a load balancer, or a NAT gateway). I’ll show how to do this below.

Terraform walkthrough

With that explained, here is the complete Terraform code you need to get a server up and running on AWS. You can get the entire thing in one file from here:

• ec2 with public IP

• ec2 with elastic IP

Here's a visual summary of what we're going to do:

EC2 with public IP example

Boilerplate terraform initialization code:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
provider "aws" {
  region = "us-west-1"
  # Optional, this will tag all resources we create
  # so we can easily find them later to delete.
  default_tags {
    tags = {
      Terraform = "true"
    }
  }
}

This is just boilerplate code you will need when using Terraform with AWS.

VPC

Now let’s create the VPC. It needs a CIDR block. The CIDR block can be anything, but I recommend using /16 as the suffix.

resource "aws_vpc" "main" {
  cidr_block       = "10.0.0.0/16"
  # If you set the `Name` tag, AWS will use it
  # for adding a name to your resource in the console view.
  # This works for some resources but not others.
  tags = {
    Name = "terraform"
  }
}

Subnet

Create the subnet and associate it with the VPC. Notice the CIDR block for the subnet is a subset of the CIDR block for the VPC.

resource "aws_subnet" "public" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.0.1.0/24"
}

AMI for EC2 instance

Now we want to create the EC2 instance and put it in the subnet. Every EC2 instance needs an AMI (Amazon Machine Image). You can get the ID for the AMI you want from AWS here: https://console.aws.amazon.com/ec2/

Or you can just look it up in Terraform like this:

data "aws_ami" "ubuntu" {
  most_recent = true
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }
  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
  owners = ["099720109477"] # Canonical
}

EC2 instance

Now let's create an EC2 instance that uses that AMI.

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"

  # assign it a public ip so we can connect to it
  associate_public_ip_address = true

  # references security group created below
  vpc_security_group_ids = [aws_security_group.sg.id]
  lifecycle {
    replace_triggered_by = [aws_security_group.sg]
  }

  # subnet to launch the instance in
  subnet_id = aws_subnet.public.id

  # simple server running on port 80 so we can verify
  # that the instance is up and we can connect to it
  user_data = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p "80" &
              EOF
}

Security group

We'll also create a security group that allows inbound HTTP traffic on port 80 from anywhere:

resource "aws_security_group" "sg" {
  name = "terraform"

  # We need to explicitly put the security group in this VPC
  vpc_id = aws_vpc.main.id

  # Inbound HTTP from anywhere
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

IGW

Create an internet gateway and associate it with the VPC

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id
}

Route table

Create a new route table and route and add a route to the internet gateway

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

Associate our public subnet with this route table:

resource "aws_route_table_association" "public_subnet_asso" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

Finally, we need to output the public IP of the instance so we can connect to it.

output "public-ip" {
  value = aws_instance.web.public_ip
}

# Optionally, output a URL for convenience
output "url" {
  value = "http://${aws_instance.web.public_ip}"
}

Try it!

Put all that in a file called main.tf. Run terraform init and terraform apply. I'm glossing over the details of how to use Terraform here, since there are other tutorials on that. After the changes apply, Terraform will print out the IP address and URL.

Hit the url that gets printed out using curl <url>. You may need to give it a few minutes for the instance to boot up.

If you get an error:

• If you get an error right away, that means everything works and you can hit your instance, but the server isn't up for some reason.

• If there's a wait before you get the error, that means you weren't able to connect to your instance at all. This could be any number of things, such as the IP you're using is wrong, or your security group or NACL are not set up to allow traffic in.

EC2 instance with elastic IP

All of the above, plus:

Request an Elastic IP and associate it with your instance:

resource "aws_eip" "lb" {
  instance = aws_instance.web.id
  domain   = "vpc"
}
# print the elastic IP
output "elastic-ip" {
  value = aws_eip.lb.public_ip
}

The end

And that's it! That is my introductory guide to networking for AWS. To close out, please enjoy this drawing of Nicholas Cage.

Back to index

We’re connecting an EC2 instance to the internet. We have learned a ton, but we still can’t connect to the internet, because we haven’t taken care of…

…security.

Here’s where we are now:

Setting up subnets, route tables etc is not enough. You also need to allow internet connections to pass through security: through your security groups, and through your NACLs. Those are two different things, and operate on different levels:

• Security groups operate at the firewall (or EC2 instance) level

• NACLs operate at the network (or subnet) level.

Security Groups

By default, your instance's security group is not set up for it to talk to the internet. You will have to explicitly set this up.

To set up a security group to talk to the internet, you need to set up rules. A security group has inbound rules and outbound rules.

• Inbound requests are requests coming from outside to your EC2 instance.

• Outbound requests are requests going from your EC2 instance to the broader internet.

Let’s look at an outbound rule that says you can connect to the internet. No inbound rules. That means that your servers can hit the internet, but no one from the internet can connect to your server.

Here’s a security group that allows outbound traffic to anywhere:

resource "aws_security_group" "sg" {
  name = "terraform"

  # We need to explicitly put the security group in this VPC
  vpc_id = aws_vpc.main.id

  # Outbound HTTP to anywhere
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

We’ll cover the options shortly. For now just know we just defined a security group with a rule using terraform, and that rules.

Here’s another security group example. This one allows inbound traffic to port 80 from anywhere:

resource "aws_security_group" "sg" {
  name = "terraform"

  # We need to explicitly put the security group in this VPC
  vpc_id = aws_vpc.main.id

  # Inbound HTTP from anywhere
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

You’ll want that one if you’re trying to put a server online.

Lets break down those fields

For the inbound rule above, we're saying that

• We allow connections only on port 80

• The server is running on port 80

• We allow the TCP protocol

• We allow connections from any IP.

The outbound rule we had seen above is even more permissive. That rule allows connections from and to any port, using any protocol, from any IP.

Note: On a toy server, you probably want that outbound rule so you can install packages on your EC2 instance. On production servers, people often don't have an outbound rule. This is so if a hacker gains access to the EC2 instance, they can't phone home. Users can still connect to their server because they have an inbound rule.

Here’s one more useful one – ssh from anywhere:

  # ssh from anywhere
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

Useful so you can log on to your box. People often recommend making this more restrictive, so only your IP address is allowed to SSH onto the server.

State your purpose

Going back to that outbound rule example. Now we can hit google.com from our server (an outbound request). But wait: the response from Google would be inbound traffic. And we have no inbound rules!

nuance incoming

Normally that inbound traffic wouldn’t be allowed, but since this is a response to your request, it is allowed. However, if Google wanted to initiate a connection, that would not be allowed.

Sometimes you will hear security groups described as stateful. This is what that means. Stateful means the response to a request is allowed, even if inbound requests aren’t allowed. You don’t need to do anything special to enable this: it’s just how it works.

A couple misc notes

You attach a security group to an instance. An instance can have multiple security groups attached to it, and you can attach a security group to multiple instances, so it's a many-to-many relationship.

Now, before we move on to NACLs, here's a quick debugging tip.

• If you try to connect to your server and the connection just hangs for a long time, the issue is probably your security group.

• But if it connects but fails instantly, it's because your server is not running.

NACLs

We won't talk too much about Network ACLs, or NACLs, because the good news is, NACLs by default allow you to connect to the internet. You shouldn't need to do anything here.

Most of the time, if you want to make a security change, you will be making it to your security group.

So why use NACLs? It's useful to use a NACL when you want to block a specific IP.

Notice that you can only allow connections using security groups. You can't create a rule that would deny a connection. That plus the fact that we use CIDR blocks to specify IP addresses, means you can't really block a single IP address using security groups. But you can using NACLs. With NACLs, you specify a bunch of rules with a priority level, and rules with a smaller priority level are applied first. You can use this functionality to block specific IPs.

Summary

• You don't need to change your NACL.

• You do need to change your security group to explicitly allow inbound connections.

• You attach your security group to one or more EC2 instances.

• If you can't connect to your EC2 instance, and the request just hangs forever, it's probably because of your security group.

Thanks for reading. In the next post, we'll put it all together, and finally talk to the internet!

Chapter 6: Terraform

Hi reader! This is part of a series on AWS Networking. The story so far: we've learned about VPCs. We've learned about subnets. We created a subnet and an internet gateway. Now we want to connect our subnet to the internet gateway. In order to do that, we need to learn about CIDR notation and route tables. This chapter is on route tables (“root tables”?).

tl;dr

1. Make a new route table

2. Add a rule to route to your internet gateway

3. Associate this route table with a subnet

Code for doing this

Longer explanation:

Here comes a packet

Someone wants to connect to your server. A packet comes in, with a destination IP address, and says "help I'm trying to go to 175.88.11.12". How do we know where the packet needs to go?

Another packet comes up and says "help I'm trying to get to 142.250.191.110". This packet actually wants to go out to the internet. How do we direct this packet?

The answer to both questions is a route table! A route table will map a destination to a target.

Route tables are mappings from a destination to a target

Route tables are set up so given a destination, the route table tells you the next step to take. It's just like if you're trying to go to a city:

In this example, the route table would look like this

The destination you're trying to get to is Mumbai, and the next step you need to go to is the airport.

Nuance! The route table doesn't tell the packet how to get to its final destination. It just tells the packet the NEXT STEP to take.

We want to connect a subnet to an internet gateway. The route table for that might look like this:

Lets break down what this table is saying.

Local or internet?

Suppose a packet comes in with a destination IP address. That IP address can be within your VPC, in which case the route table should route it locally

The route table tells the packet where to go next. If it's any other IP address, it should route it to your internet gateway

What does the route table for the logic above look like? Well, to start we need two routes:

1. If our EC2 instance wants to hit an IP address inside our VPC, we route that request locally inside the VPC

2. if it wants to hit any other IP address, we route that request to the internet, through the internet gateway

Here's what the route table for the logic above would look like:

You’ll notice we are using CIDR notation to specify a range of IP addresses (quick tutorial on CIDR here). The first rule says, any IP address that starts with 175.88 should get routed locally. The second rule says, all IP addresses should get routed to the internet gateway.

You can add routes for specific IP addresses,

but typically you will create routes for CIDR ranges.

Route tables pick the most specific route that matches

The route table in our example had two routes

Notice that one of the routes matches any IP address

while the other one matches only IP addresses in your VPC

So if a packet comes in with a local IP address, technically it matches both routes. Where will it go? Route tables will always try to match the most specific route.

Psst… how is the matching done? Using something called a Genmask. This isn't need-to-know information, so I won't describe it here, but now you know the word and can look it up if you want to.

The main route table

Route tables are created at the VPC level.

This means VPCs can have many route tables. Every VPC comes with a route table, called the main route table. This route table will have a route by default that maps your VPC's CIDR range to the local target. Typically, you want to leave this as is.

Subnets and route tables

All subnets are associated with the main route table by default. Suppose you have two subnets, both private. You're trying to make one of them public, i.e. giving it a connection to the internet. You do this by attaching it to an internet gateway. To do that, you need to add a route to your route table. Suppose you add a route like this to your main route table:

This works, and one of your subnets is now public. Unfortunately, all of your subnets are now public! Since you made this change on the main route table, and all subnets are associated with the main route table by default, all your subnets are now public.

To make a subnet public, you instead want to make a new route table and associate your subnet with that route table.

But there's a gotcha here. Subnets can only be associated with one route table at a time, so now your subnet will no longer be associated with the main route table. Remember that the main route table had that route for local traffic. Now your public subnet can no longer handle local traffic! If you want the subnet to handle local traffic, you'll have to add a route to your new route table to handle local traffic.

Terraform

# Make a route table with one route,
# which matches any IP address and
# sends it to the internet gateway.
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

# Associate the route table with the subnet.
resource "aws_route_table_association" "public_subnet_asso" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

(Excuse the lack of syntax highlighting — Substack doesn’t have that)

Here is the Terraform code for everything we have learned so far, including making a VPC, subnet, etc: Github gist

Bonus command line tip!

If you are SSHed into an EC2 instance, you can see the route table by using the route command. You can also do netstat -r, which just calls route -e in the background.

Summary

Phew! You finally have a route in place, and your subnet is connected to the internet gateway. Here's what we have so far:

Bullet points

• To connect a subnet to the internet, you need a route.

• Routes are defined as rules in a route table.

• A route table matches a destination to a target.

• Destinations are usually specified as CIDR ranges.

• Route tables are created on the VPC level. Each VPC comes with a default route table called the main route table.

• You probably want to create a new route table and associate it with a subnet. Subnets can only be associated with one route table at a time.

Your subnet is now connected to the internet, but you still need to set up your security rules to allow traffic in. We’ll see that next!

Chapter 5: Security groups

Hello AWS adventurer, and welcome back to Chapter 3! So far we have learned about VPCs, internet gateways, and subnets. We have learned that a VPC is our own private network on AWS. We have learned that we need to create instances inside subnets. And finally, we have learned that to connect our servers to the internet, we need to connect our subnet to an internet gateway.

In order to connect our subnet to an internet gateway, we need to use a route table. And to use a route table, we need to know about CIDR notation. So this chapter is about CIDR notation.

What is CIDR notation? Well, to recap, this is an IP address

Instead of a single address, CIDR notation lets you specify a range of IP addresses.

This particular notation is equivalent to saying

Let's dig into what this notation means. In an IP address, you have four numbers. Each number is 8 bits, so it can be a number from 0 to 255

Lets look at the CIDR range 175.88.0.0/16. The 16 means the first 16 bits are fixed, which means the last 16 bits can change.

You can equivalently say this as the last two numbers can change,

or replace the last two numbers with x's,

or say this range

All of those are the same.

That /16 is called a CIDR suffix.

Here's another example

This is a /30, so the first 30 bits are fixed, and the last two can change. Note that it's convention that the numbers that can change are written as zeros, but you don't need to.

And, of course, when using CIDR notation, the last number can be a zero as well

In that case, we're talking about all IP addresses. This is more commonly written like this

You may see this notation in a route table. For example, if you have a route table where a connection to any IP address gets routed to an internet gateway, you would write it like that

(We'll talk about route tables in the next chapter, so don't worry if you don't understand this image).

VPC CIDR Ranges

BTW, each VPC you create will have a CIDR range. That means it's assigned a range of IP addresses, and the resources within that VPC have IP addresses that are somewhere in that range. For example, say the range for this VPC is 172.98.0.1 to 172.98.255.254. Any resource in this VPC will have an IP address somewhere in that range. That means that every resource in this VPC will have an IP address that starts with 172.98.

Subnets have their own CIDR ranges, and since each subnet is inside a VPC, its CIDR range is within the VPC's CIDR range.

For example:

Notice the /24 is a smaller range than the /16, even though the number is bigger.

That's all there is to it! Explanations for CIDR notation can get complicated, but at its heart, we just use it to specify a range of IP addresses.

In the next chapter, we will learn more about route tables. We’re about halfway through our journey, at the end of which, we will be able to connect an EC2 instance to the internet. Yes, it’s a long journey. But I'm glad to be with you, Samwise Gamgee.

Summary

• CIDR notation lets you specify a range of IP addresses.

• Each number is 8 bits.

• The bigger the suffix, the smaller the range.

Chapter 4: Route tables

In the last chapter, we talked about VPCs, and how a VPC is a private network inside AWS. It's like a big bubble of protection that we've built around our instances.

There are other people using AWS, and some of them might be bad people who want to connect to our instances and steal our data. But our instances are in a private network that no one else can get into.

If you haven't read the chapter on VPCs,

Read the chapter on VPCs

The benefit of having a protective bubble is that no one can connect to our instances. The downside of having a protective bubble is that no one can connect to our instances! We're building a website, after all, aren't we? How's anyone supposed to connect to our servers with this darned bubble in place?

We need to poke a hole in our bubble so we can talk to the internet.

Internet Gateways

An internet gateway is like a hole you poke in your VPC so you can talk to the internet.

Simple.

End of section.

Terraform code:

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id
}

Now you have an internet gateway (aka an IGW, pronounced "igwoo"). Here's what we have so far:

Here's what the internet gateway looks like in the AWS docs:

Let's zoom in:

Zoom in some more:

Beautiful. Like the Gateway of India, but online.

So now we have an internet gateway (IGW), we can connect to the internet, right?

No! Not even close! Muahahahaaaaaa

Let's zoom out. First we need instances.

But before that (so zeroth), we need a place to put them in. See that green box labeled subnet?

Subnets

You can't connect your EC2 instance to an internet gateway directly! Only a subnet can be connected to the internet gateway. You need to put your instance inside a subnet, and connect that subnet to the internet gateway.

A subnet is a way to group your instances. It's sort of like tagging them. Here I have four instances

and now I've tagged them with subnet A or subnet B

To connect an instance to the internet, you need to put it in a subnet that is connected to the internet. Notice I said "put it in a subnet", but the subnet's not like a basket you're putting your instances into. It's just a tag, or a connection. Or an assignment.

Public and private subnets

By default, subnets can't talk to the internet. Subnets without a connection to the internet are called private subnets. Subnets that have a connection to the internet are called public subnets. To get your instance connected to the internet, one of the steps you need to take is assign it to a public subnet.

Sidebar: Why do we need to put our instance in a subnet? Well, we can't just put an instance in a VPC, because remember, the VPC spans all the availability zones in a region. To create an instance, we need to specify which availability zone we want to create the instance in. That way, we can create instances in multiple availability zones, to guard against outages. VPCs are region-scoped, but subnets are availability zone-scoped. To assign our instance an AZ, we assign it a subnet.

Now we have something like this

And to connect to the internet here's what we need:

Everything is ready, but to connect the subnet to the internet gateway, we need a route.

Terraform code for creating a subnet:

resource "aws_subnet" "subnet_name" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.0.1.0/24" # see next chapter
}

Routes

A route is a connection from your subnet to the internet gateway so that your subnet can talk to the internet.

We need to create a route from our subnet to the internet gateway. To do that, we need to learn two new concepts first: CIDR notation, and route tables. The next chapter will talk about these stimulating concepts.

Summary

• An internet gateway is like a hole you poke in your VPC so you can talk to the internet.

• You can't connect your EC2 instance to an internet gateway directly. You need to put it in a subnet, and connect that subnet to the IGW.

• Subnets without a connection to the internet are called private subnets. Subnets with a connection to the internet are called public subnets.

Chapter 3: CIDR

One reader added some fascinating insights:

Security groups were the original solution to the second problem (that everyone's servers were in the same network).

The address conflict problem transformed into another problem after VPCs. After everybody got default VPCs, companies went from conflicting with each other to conflicting with themselves. Now they have a bunch of instances with the same IPs, in different VPCs, that still need to talk to each other… a problem that needed double-sided NAT.

A big part of the first problem (IP address conflicts) was actually that AWS was about to run out of IP addresses to issue to customers.

Back then, every instance got a public address because there was no other way to ssh into your instance. Because public addresses are so expensive and limited, this was a scaling cliff for AWS, and passed on as an unnecessary cost to customers. VPCs let you use a single bastion with a public IP to access the rest of your hosts.

This series of posts will get you up to speed on AWS networking, so you can get an app up on AWS. Every post is meant to be easy to understand, and spends time really explaining the concepts, so you understand how things work, and can adapt them to fit your use case. Each post is full of illustrations that I hope will brighten your journey as you learn AWS networking. Good luck and happy reading!

Read Chapter 1: VPCs

Index

1. VPCs

2. Subnets

3. CIDR

4. Route tables

5. Security groups

6. AWS in Terraform

In this section, I talk about why VPCs were invented and how they work. This is critical to understand because almost everything you do in AWS will happen inside of VPC. If you don't understand VPCs, it will be difficult to understand any of the other networking concepts.

If you're reading this, maybe you have one of these

and you just found out that to put your app on AWS, you need all of this:

And you have no idea what VPCs, subnets and so on are.

I'll help you learn about all those pieces. A little about me, I’m a long-tailed duck, and I run a business selling phones to hackers, called Blackhatberry. Now let's get started.

This is the story of VPCs (Virtual Private Cloud)s, our first big topic. Many moons (and suns) ago, some AWS engineers were sitting in a room. They had a serious issue.

"Guys, lets talk business. Why aren't more companies moving to AWS?" they said.

"Maybe because all instances run in a single shared network, which means users can access each other's instances, and see each other's data," someone said.

"Maybe because it's hard for them to move their existing servers to AWS, because of IP address conflicts," someone else said.

"Wait… what are IP address conflicts?”

“And existing servers? Shouldn’t they be moving to our servers?”

IP address conflicts

This is the first reason people weren’t switching to AWS. Here's what I mean by IP address conflicts. I own a bunch of servers for Blackhatberry. One of them has the IP address `172.98.0.1`. Now, my neighbor also has a server for her business. She loves my ip address. “Ah, 172.98.0.1, what a beautiful destination,” she says. So she copies my address! Now we both have servers with the same IP address!

Sidebar: You can find your local IP address using `ipconfig getifaddr en1` (works for Macs for wireless internet connections).

Now you're thinking "so what?". And actually... you're totally right. Even though our servers have the same IP address, they are in different networks, so it's not an issue.

But here comes trouble. Because we both want to get on AWS. But if we have two servers with the same IP address on AWS, that's a problem!

Bam: IP address conflict. Every server in a network needs to have a unique IP address for the same reason that every house in a city needs to have a unique address. Otherwise, if someone has a package, they wouldn't know which house to deliver it to.

Now I know what you're thinking. Why don't you just get new servers in AWS? Why connect your on-prem ("on premises") servers to AWS? Isn't the whole point that you're moving to AWS?

And sure, we can afford to do that, but there are companies with dozens of on-prem servers. Migrating everything to AWS is simply not an option for them. They need functionality so they can create new servers in AWS, but also connect their existing, on-prem servers into the same network. And this does not work when everyone is part of the same network, because of IP address conflicts.

This was a huge problem for AWS! I mean, see how serious these engineers look:

This IP conflict issue meant people with on-prem servers had no easy way to gradually move to AWS. Think of all the potential customers they were losing!

I'm giving you this background so you can understand why VPCs were invented. IP address conflicts weren't the only issue. In AWS, everyone's servers used to be on the same network, which meant if you were careless, it was easy for anyone to connect to your server and look at all kinds of sensitive data!

For both these reasons, Amazon needed to give each customer their own private network, instead of having them all on the same shared network. And so VPCs were born.

Why do they call it a VPC if you can't see it!

So there are two problems we're trying to solve:

1. IP address conflicts

2. The fact that users can access each other's instances because they're in one big shared network.

Remember: duplicate IPs were totally fine when my network and my neighbor's network were separate. What Amazon needed was a way to give each person their own private network, but inside AWS. That way, they could bring their IP addresses with them, and they wouldn't conflict with anyone else's IP addresses.

Maybe you're wondering, "why can't we just change the IP addresses so all machines have a unique IP address?" Well, in networking, you set up some things based around specific IP addresses (I'll get to exactly what stuff later), so that idea would require a lot of work in practice.

Separate networks would also solve the security problem.

This is the big idea behind a VPC: everyone gets their own private network inside AWS.

By the way, why am I spending so long on VPCs? Isn't this post about putting one of these

on one of these?

Two reasons.

1. Because everything we will build happens in a VPC, so it's the starting point for things.

2. Because a VPC is not something you can see, and I like to visualize my internet architecture. Other people visualize it in a way that's really confusing for me, and I want to make it less confusing for you.

I have read guides (such as the AWS docs) where people visualize a VPC like this. First, they'll say, "Oh yeah, I created a new VPC with four subnets inside it, in two availability zones".

And they'll draw an image that looks like this:

But less pretty obviously – this is what theirs look like:

(Taken from the AWS VPC docs)

Now,

• a region is a place you can go to,

• and an availability zone has data centers you can walk inside, that hold lots of servers.

Both of those are physical places. But what is the VPC? Is it a big tarp that sits on top of the data centers? Is it a dark fog? Is it a general feeling of unease that blankets the region, as all the data centers play Radiohead's "Fitter, Happier" on repeat?

…what is it?

What is a VPC?

We've talked about why AWS needed VPCs, and the idea behind VPCs, but how are they implemented? How do they actually work?

Your instances in AWS always run inside a VPC. But in real life, of course, your instances are just running on servers in AWS datacenters.

You may have many instances running on many different servers. How does Amazon connect these instances, across different servers, into their own private network? It uses something called the mapping service.

Mapping service

Suppose I have an instance A on server 1, and I want to talk to another instance B on server 2.

All instance A knows is the IP address for instance B. It hits the mapping service with that IP address. The mapping service then checks what VPC instance A is in, finds the instance with that IP in that VPC, and forwards the request to it.

That "in that VPC" part is important. The mapping service makes it so me and my neighbor can both have instances on AWS with the same IP, but when I hit that IP address, the mapping service will connect me to the instance in my VPC, and when my neighbor hits that IP address, the mapping service connects them to the instance in their VPC.

The mapping service is what isolates our servers.

The mapping service is what ensures that we can never connect to each other's instances. Through the mapping service, all my instances are connected together, and they can have any IP address I want, because it's like they're namespaced to me. The mapping service is what creates the private network inside AWS for me.

So when you think VPC, picture a service that connects all these instances together.

Going back to this image, we can now understand what it means:

That VPC box just means the scope of the mapping service. A mapping service can connect EC2 instances that are on servers in different availability zones, which is why the VPC is overlaid over the two availability zones. But the mapping service can't connect instances in different regions, so the VPC doesn't span regions.

Today everything happens in VPCs. Your instances are always in a VPC. Everyone gets a default VPC when they open an AWS account. We no longer have the issue where users can access each other's instances, and we don't have the IP collision issue either.

So we find out that they don't play "Fitter, Happier" in the data centers after all. Maybe it's just a recording of Jeff Bezos singing "Money" by Pink Floyd.

Terraform code

Throughout this guide, I'll show you how to create AWS resources using Terraform. I find Terraform easier to follow than point-and-click on the AWS console, because you can just copy the code and run it.

Here's the Terraform code to create a VPC:

resource "aws_vpc" "main" {
  cidr_block       = "10.0.0.0/16"
}

In any Terraform file, you'll also need a couple of boilerplate blocks for terraform and provider. The full code listing is here. You can apply this code to create a new VPC. Ignore the cidr_block part for now, I'll discuss CIDR in more detail in a future section.

Summary

• In AWS, every customer has their own private network called the VPC.

• Without private networks, we run into IP address collisions.

• Without private networks, everyone is on the same network, which is really bad for security.

• VPCs are implemented using the mapping service.

Chapter 2: subnets

P.S. A reader provided me with some extra context on AWS VPCs. You can read that here.