Microservices are all the rage these days. Luckily underneath the hype, there are some great use cases for them. If you’re splitting up a monolith codebase into smaller specialised chunks, extracting a long-running queue to its own system, or even using particular pieces of code in a different programming language. You always have to solve one architectural problem. How do I make these things talk to each other?

Your first reflex might be a collection of REST API's. Put them in a network with filtered ip’s, or any other way to secure it, and call it a day. But if we take a step back and look at this setup, is REST really the best way to go about it?

Don’t get me wrong, I love REST. But in this case, should we be thinking in terms of resources? If we take a system that crops avatars for example. You would get something like this

GrpcPatch

This doesn't really feel right. We aren't dealing with resources here but more with actions on data.

Fortunately, we aren’t the first ones to ponder on this.

Some of you might have had a shiver run down your back when reading that title. RPC is commonly linked to older systems and even worst: linked to SOAP (shiver nr.2).

The reason you might not have great experiences with these setups is, ironically, for the same reason we described above. They have been misused to fit the architecture they aren’t designed for. RPC is actually perfect for the use case above.

For the people not familiar with RPC, let me give a quick overview: It stands for the Remote procedure call. And is, as the name suggests, a way to call procedures on a remote system. The REST call we made earlier would look like this:

GrpcPost1

GrpcPost2

You treat your external API as an internal API. Parameters for the endpoint are attached in the body of the call. And that gives some weird results sometimes. In the case of RPC, Verbs don't really matter. If you want to fetch a user for example, it could look like this:

GrpcPost3

GrpcGet

If you’re an avid REST supporter and never worked with RPC before, this all looks very strange. Maybe even dirty. I personally don’t like public RPC API’s, but in very specific cases they might be useful. If your API is more like an application that does actions on data, instead of managing data, it can make sense. (Just remember that this is not an excuse for letting your internal architecture bleed into your API).

Knowing this, we can look at our microservices in the same light. Here we have small applications that might benefit more from the RPC style of calling methods than the REST style of altering resources.

A nice end goal for our microservices might be to use them in our code, without knowing that they are remote. Let me better explain this with an example.

GrpcPhp.php

It would be great if we could use code like this to communicate to a remote service. The Mailservice would be an abstraction that takes the method SendToUsers dynamically from the microservice.

In that way, when you update the API of the microservice, you would not need to update the Mailservice code. The client would automatically have access to the new methods.

Of course, that would take a lot of work to create. You would have to write some sort of transformer, create something that can login to the service, and eventually POST the data…

There is another company with the same issues as us. Ok maybe not exactly the same issues as us, cause they are maybe a bit bigger. Google created a system called Stubby that connects most of their services together. When Stubby got a rewrite, Google decided to create a new system. Not closely matching their infrastructure and even made it open-source. This new system is called GRPC.

GRPC is an RPC framework that does a lot of heavy lifting for you. First of all, it handles all the load balancing, tracing, health checking and authentication functions you would otherwise have to write yourself. It can even go full async. But that is not even the coolest part. It also writes all the code for the connection.

If you want to execute methods on a remote system as if they were present in your current system, you will have to transform the data. You have no idea what the server is expecting. The server on the other side would want to have that data serialized as well. This is also a great place to add some validation to that data.

To tackle serialization and validation Google created something called protocol buffers. These protocol buffers rely on proto files that tell them what data is coming in and out. This system is somewhat comparable to systems like Swagger and Blueprint. Unlike these, proto files are less human-readable and look a bit more like code. And pack a lot more functionality than just documentation.

Here is an example of one of those proto files:

GrpcProto.proto

As you can see they have a C/Java style and more importantly, are strictly typed. The numbers that are assigned to these variables are ids for the system itself. Please note that these files are allowed to have a bit of logic in them making them very flexible. You can look at them as value objects.

Once you have created the Proto file, the real magic happens. Now you are able to generate the connection between the systems. I say generate because it is exactly like that. GRPC can now automatically generate an SDK that you can import into your client and server. This SDK uses the mapping to act as regular methods call in your current codebase.

You can find some examples of generated code on the GRPC github

Why would I want proto buffers over something like JSON? 

We now know on a high level how GRPC works, but we still haven’t talked about how the data is actually transmitted. The downside here is that unless you dive really deep, this will all be a bit black box-ish.

Protobuffers aren’t really designed to be human-readable, they are compressed binaries. I know this might be a huge turn-off, but it does have its advantages. These come mainly in the form of its small footprint and speed.

Yogesh Shinde posted an article on Dzone on the comparison between JSON and proto buffers that suggested that the latter are almost double as fast.

Bruno Krebs’s article Beating JSON performance with Protobuf even sported cases that were up to 6 times the speed.

Everything comes at a price, and GRPC is no exception. The biggest downside is obviously the black box connections. They are hard to debug and you’re not ever really sure about what’s transmitted.

The generated code is also a bit of a double-edged sword. Lots of people have had some bad experiences with generated code (for good reasons). But to be honest, I feel that in this case, it’s not that bad. The generated code is mostly value like objects that sometimes are little more than getters and setters and it does allow you to generate SDK’s in loads of different languages with a single click, so that is a decision you have to make for yourself.

I also found the set up a bit hard. In the case of PHP, they suggest you install using PECL what is not really ideal (There is a composer option as well). You also need to run a Node server so that might not jive all that well with your current setup.

This might all seem a bit daunting to set up if you only have 1 or 2 microservices in your domain. But if your product is a collection of different microservices, you might save some time and performance with this technology. Especially if there are a few different programming languages mixed in.

But be sure to create a pilot project before you take the plunge.

Microservices can solve a lot of architectural problems, and sometimes create a few fun new ones. A big problem however is connecting these services to each other.Can GRPC lend a hand here?

Connect microservices with the help of GRPC

So an API, that's just output of some JSON code right? No problem, my framework does that automatically.

Or if you just want an overview on some best practices, conventions and nifty ways people have done stuff before, this post has you covered.

We won't go into much detail about source code here. This article is language agnostic, whether you use PHP, Ruby, Velato or some obscure Javascript framework you can happily follow along. Please note that not everything here is law, some parts (maybe everything) are open for discussion. Lastly: some of this may be pretty obvious, you are probably right. But I rather cover all the bases.

Application Programming Interface, or as the cool kids call it: API, is a collection of endpoints to interact with an application. You have internal and external APIs. This post is all about external APIs (more specific, external web APIs). Some people like to think about APIs as a wrapper around a database, this is however mostly not the case. But they tend to serve (mostly) the same purpose: interacting with data.

APIs can be widely different in architecture (XML-RPC, REST, SOAP, ...) and data structure (XML, JSON), but it's very important that they stay as consistent as possible. Today we're gonna focus on REST.

Stands for Representational State Transfer (don't worry, there won't be a quiz) and is the most commonly used (and modern) architecture for handling APIs. REST doesn't rely on the coupling to the rest of the application like protocols as SOAP tend to do (do note that REST is not a protocol) making it fast and easy to implement.

The basics of REST aren't that hard to grasp, although there is a lot of misunderstanding on the internet about it. Rest is a resource-based architecture that is stateless. This means that a REST resource (the data) can't have sessions nor cookies. The data itself is represented in media types like XML and JSON. It's a common pitfall to think that RESTful APIs are JSON only.

REST commits to a uniform interface. In most cases, this is HTTP as they fit nicely together. However be careful, another common pitfall is to think that HTTP and REST are interwoven. REST isn't necessarily connected to HTTP even though in practice it usually is.

So what does that exactly mean, committing to a uniform interface? This is easily explained with the HTTP protocol:

Hypertext Transfer Protocol (another acronym, this post would be a great drinking game) is something you probably already know. You are using it right now, as you are using it in your browser to surf the Internet (a relatively new hype).

HTTP is request/response-driven, this means that response only is given as a request is made. There is no constant stream of data (sockets).

In the case of REST, we are mainly interested in its interface. In HTTP the client sends verbs in the header of the request, the server responses with HTTP codes in its response header. Now, this is one of the hard parts about building an API, what verb fits with what action and/or what response code should be sent. As far as the verbs go, only the edge cases are difficult, the response codes on the other hand are often open for discussion. So let's have a quick look at some commonly used ones shall we (please note that these are the most used ones, there are more):

| Verb | Return HTTP Code | Example of url |

It kinda does what it says on the tin. It GETs a resource. It can fetch individual resources or a collection (We'll take a closer look at how this is displayed later in the post). On success, this endpoint returns a 200 (see HTTP codes) with the resource (or collection of resources) in its body.

This verb asks the server to accept the resource included in the post. You mainly know this outside of APIs as forms. It's commonly used to add new resources to a collection. For example, If you were to post to animals/wombats you would create a new wombat, not a new animal. If the endpoint creates a resource that isn't covered in the API (not everything needs an endpoint) it should return a 204 No ContentHTTP status code (success, empty response), or a 201 Createdresponse which should contain the created resource and optionally a location header where the new resource can be found.

Now, what do I mean by "creating a resource on an endpoint that isn't covered in the API"? This would be an endpoint that has no reference GET endpoint. Simply put, something you only want to send to the server but never return to the client.

The PUT verb is like the POST verb used to send data. The main difference here is that POST creation is done on the collection, PUT creation is done on a resource URI (POST: animals/dogs PUT animal/dogs/1). PUT can also be used to alter a resource. Executing the same PUT request multiple times should only create or alter the resource once: its action should be idempotent. If you do this then please be sure to inform the client of what happened, nothing is worse than getting non-sensical HTTP codes (except the new Star-Trek movies). So if you just update the resource, you should return a 200 or 204 HTTP code and an empty body. If you end up creating a new resource, you just follow the POST flow.

This one is a bit special, It's often confused with the PUT verb as it (mostly) performs the same function: updating resources. So why have two verbs that do the same thing? The answer is, they don't and you shouldn't probably use this verb (unless you really know what you're doing). According to the RFC, a PATCH request should contain instructions on how to partially update a resource.

You can find more info (and examples on how a PATCH request looks) here: REST API - PUT vs PATCH with real-life examples

Update: William Durand informed me that you should always add the header Content-Type: application/partial-update-JSON when you do a partial update through PATCH. More info can be found here: media type for partial JSON updates?

Another simple one, it deletes stuff. This needs to point at a specific endpoint (like a PUT) and returns a 200 HTTP code.

It's very important to have a consistent API, a huge part of this is the consistency of your URLs. Don't worry, if you just follow a few simple guidelines you'll be alright.

First of all, it's best to use nouns instead of verbs. This is a simple way to enforce consistency in your naming scheme. Secondly, you should always use the plural form. so use animals/dogs instead of animal/dog. Thirdly you should think in terms of your API, not in terms of your database. Your client doesn't need to know about the structure of your data model.

A very important part of an API is informing clients about what just happened. In HTTP this is solved with HTTP codes returned in the header of the response. We have seen a couple of them appear above, so what are they? Well, they are indexes to a list of commonly used responses a server (or teapot) could give.

They can be found here: List of HTTP status codes or here in a more academic way. As stated before, sometimes it's hard to see what code fits what use case. These are often fun points of discussion while stargazing on a first date.

Hypertext As The Engine Of Application State (acronym, drink) is a big part of REST. Its purpose is to add discoverability to your API. You can compare this to hyperlinks on regular websites. For example: imagine a Twitter feed, it consists of a long list of tweets sent by users. if you click on a tweet you get a detailed view of the tweet with all its responses. This is comparable with the HATEOAS structure. You attach a link to each entity in a collection, this link points to the endpoint of the specific resource.

Why is this handy? In theory, this allows dropping separate documentation. You don't have to document all your endpoints, the endpoints provide them themselves. In practice, I would advise to always provide separate documentation. A great workflow here is API Blueprint documentation with automated Dredd testing on top of it to make sure your documentation is never out of date.

This is the part things get a little fuzzier. Not only are there different content types out there (XML, JSON), even those tend to have different structure standards. This part of the article is more about what's out there, instead of what you should use. The most important thing you should remember is the scope of your application. Even though it's good practice to return both in XML and JSON (through the Accept header), it can prove to be beyond the scope of your API to do so.

As most people prefer the JSON format (as do I), we shall take a look at the different structure standards (loose term). People tend to get pretty defensive on what standard they pick, this happens because when you pick one, you're stuck with it. You can't just decide to adopt a new structure, even in a new API version, I would not advise it.

Json api

This is my favourite, so it's first. I've taken the liberty of trimming the example found on the site (http://jsonapi.org/). As you can plainly see, this includes lots of data. On closer inspection, you might notice that the "real" data is kind of small compared to the entire response (this is a very verbose example). This is due to the very heavy usage of HATEOAS.

The response starts with a links object, this is a nice way to implement pagination. It's pretty easy to write a recursive function to consume this endpoint (and you can reuse it for all endpoints).

Then follows the data object, which contains an array (if this were an endpoint to a specific entity, it would not use an array) to display the different articles (in this case only one).

The relationships are pretty self-explanatory. these are HATEOAS links to related endpoints.

Jsend

Simple and to the point, what's not to like? Well, there is no HATEOAS, that's kind of a bummer. Still, if you are making a small application (or maybe some AJAX calls), that would not be disastrous. As stated before, it all depends on the scope of your application.

OData

Now you might look at this and think: Wtf... And you would be right. This is difficult to understand standard and is probably not something you wanna use as it's not really compatible with REST. OData requires endpoints that get created dynamically (called out-of-band information). it also demands that you create an endpoint named /$metadata to decipher the metadata in the response.

This Structure seems a bit out there and requires not only some extra work to implement, it also requires some really great documentation.

I would only recommend it if you really need it, and know what you are doing. (Even then HAL would still be a better option)

Looks familiar? It's basically OData with HATEOAS support. I would choose this over OData (and not only because 2001: A Space Odyssey is one of my favourite movies).

As stated before, These are just a few options. It's not a crime to adapt these to your business cases or create a brand new standard. Maybe combine JSend with HATEOAS?

As you guys know REST is stateless (no sessions or cookies for you). So how can we limit users so they only can access the content they are allowed to access? No worries, smart people have already found some smart solutions to this problem. Again which one to choose is up to you and your application (even though OAuth2.0 is currently preferred)

This one is the easiest one to implement. It's the standard HTTP way of authentication. That said, it's also the least secure option out there (never use this over non SSL/HTTPS connections!).

To log a user in, the client just needs to send a Basic header with a base64 encoding of username:password.

That's it. Now every sequential call you do, you just have to send that data with it.

A JWT looks like this: aaa.bbb.ccc. It's just a long string divided into 3 sections, separated with a dot (.). The first section is a header, the second one is the payload and the last one is the signature. For more information about how to build token, I recommend reading this guide.

The basic JWT flow goes like this: You send your credentials to a login endpoint. The endpoint returns a JWT token with your data in it. Now you just have to send that token with every request like this:

Auth

Most JWT's have expiry dates, This kind of authentication method has no refresh system. When it expires you just have to request a new one.

This is the most popular Authentication method. This is mainly due to its security and flexibility. That said, it can be a little bit of PITA to set it up. Covering the entire OAuth spectrum would take us too far I'm afraid. I'll just cover a very simplistic practical flow.

In its most simplistic form there are some similarities between JWT and OAuth. They both request a token in the same way and they both get a token returned (In this case obviously an OAuth token and not a JWT).

OAuth requires that you send the token with it on each request as well:

OAuth tokens always have an expiry date, when this happens you have to use the refresh token (it's part of the response you get when receiving your OAuth token) on the refresh endpoint to get a new OAuth token.

This is a very simplistic representation of retrieving an OAuth token, I'm afraid the differences between OAuth and JWT would take us too far. I encourage you to read more about it here: http://oauthbible.com/.

The older your API gets, the more prone your API is to change. How we deal with this change can be tricky for end users. You can't just remove or change endpoints, there are probably some people that rely on the old structure of your API. You don't need to support older versions of your API indefinitely, but it would be common decency to give people time to migrate to the new structure.

As you can imagine, everyone has an opinion on how versioning should be done. Let's look at a few common ones.

This looks like the most obvious solution, but it's not necessarily the best one. This way of versioning requires some foreknowledge and planning. You have to design your first version of the API with the possibility in mind that future versions of the API are coming. This sounds obvious, but API versioning is commonly an afterthought.

A URL based versioning URL might look like this:

Urb based versioning

This way is commonly described as the "best" way. Of course the reality is that there is no best way. And it all depends on your use case (can't stress that enough).

Header based versioning is done trough, well you guessed it, a header. More specifically, the Accept header. Looks like this:

header based versioning

If no version header is given, you simply return your latest version.

This is of course the tip of the iceberg. We just had a bird's eye view over API development. The key thing to remember here is that almost every problem with APIs has occurred to someone before and it's important to be aware of what the possible directions are with every crossroad you encounter. Just don't forget that there is no such thing as "the one true way".

Phil Sturgeon's great Build APIs you won't hate

So you want to build a web API. You have to start somewhere, why not here

Frederick Vanbrabant's

delirious rantings

Connect microservices with the help of GRPC

A bird's eye view on API development