Developing TLS in Hyperledger Fabric

Introduction

Secure communication between components in Hyperledger Fabric is done in TLS. Examples are client application requesting endorsement from selected peers, a Fabric CA Client registering users in a Fabric CA Server, etc. As part of the TLS process, certificates are used for authentication. This is to ensure that one is talking to the right party but not anyone or anything else.

It is a challenge for some beginners when they first see TLS. Besides the lengthy path of certificates, the adoption of digital certificates in both TLS and the identity elements in Fabric always causes confusion. As we will see later, they are two independent parts, operating separately and for different purposes.

The purpose of this article is mainly about how TLS is implemented in Fabric. We first provide some basic information about TLS as a foundation for understanding TLS in Fabric. Then We explore the three options when using TLS in a fabric network, and demonstrate how these options are implemented. Finally, there are several ways to implement the infrastructure of TLS CA, and we will inspect three samples provided in fabric samples and tutorials.

TLS Basic

This part provides some foundation information about TLS. Feel free to skip this if you just wish to take a look at TLS in Hyperledger Fabric.

Overview

The main purpose of TLS is to secure the communication between two parties. This includes endpoint authentication, ensuring one is talking to the right target, and message security such that message is encrypted and authenticated.

Here is a bit of history about TLS. Evolved from SSL, TLS 1.0 was released in 1999. With a short life of TLS 1.1 in 2006, TLS 1.2 was released in 2008, which is the most common version used in today’s Internet. Fabric supports TLS 1.2 prior to v2.2. TLS 1.3 was released in 2018, and is supported in the latest released Fabric v2.2 (July, 2020).

Review: Symmetric Cryptography and Asymmetric Cryptography

TLS uses both symmetric and asymmetric cryptography. Here we first make a quick review on them.

Symmetric cryptography: two parties are using the same key (always called shared key) to encrypt / decrypt and authenticate the message.

Asymmetric cryptography, always known as public key cryptography, is different from symmetric one in that it involves more than one key. In this case each party has a key pair. This key pair is mathematically related in a way that messages encrypted by one key can only be decrypted by the other key of the key pair. In practice use, one key of the pair is designated as private key, kept by the key owner, while the other key is designated as public key, known to the public. The name public key cryptography comes from this designation.

While at the first glance asymmetric cryptography seems more advanced, it is in fact very costly in operation. In TLS context, message encryption is done with symmetric cryptography, while asymmetric cryptography is used for key exchange / generation, and for certificate (identity) verification.

Symmetric cryptography and asymmetric cryptography

TLS process overview

In this session we will walk through the TLS process. We reverse the sequence by first inspecting the last stage (bottom) and then moving upward. This helps us understand why things are working in that way.

Establishing TLS communication between two parties

Message communication with symmetric cryptography

Majority portion of communication is done after shared keys are available for the two parties. This is the final stage after both sides obtain the shared keys.

  1. With the same shared secret, now Alice and Bob can communicate their messages safely.

One very logical question is, how can both parties come up with this shared key? It does not make sense to preload shared information between different parties due to security concerns and scalability issues. Instead we let both parties generate them dynamically. It is where asymmetric cryptography plays a role.

Key exchange / generation with asymmetric cryptography

The objective of asymmetric (public key) cryptography here is to let both parties agree upon some shared secret, and from which shared keys are derived by each party individually. There may be some variations depending on the implementation, and here is one possible process. Note that the final step is what we had in the previous session.

Assuming Alice wishes to reach Bob for TLS communication

  1. Alice sends a hello to show her intent to communicate with Bob with TLS.
  2. Bob sends out his public key and Alice receives it.
  3. Alice encrypts some secret using Bob’s public key.
  4. Bob obtains that secret by decrypting the received message with his own private key. Note that now Alice and Bob have the secret, which is only known by them, but not by others.
  5. Applying some mathematics on this secret, Alice and Bob come up with some shared material, which is again only known to Alice and Bob. The shared material is being used for symmetric key encryption.
  6. With the same shared secret, now Alice and Bob can communicate safely their messages.

Now we have used public key cryptography to let both parties have the same shared secret. But there is still something to solve in step 2: how can Alice know she is talking to Bob, or more technically, how does Alice know that the public key received in step 2 is really from Bob but not from someone else? This involves endpoint authentication to make sure Alice is talking to Bob but no one else.

Authentication with Certificate

The final challenge is to identify whether Alice is Alice and Bob is Bob, that is, in step 2 of our previous process, how Alice knows the public key shown by Bob is actually Bob’s. Technically speaking, a public key belongs to Bob means that Bob and only Bob has the private key which forms a valid key pair. And here we have assumed Bob’s private key is not compromised, and only Bob is holding this private key.

Certificate is the way TLS handles authentication. For simplicity we can see certificates as an association of a public key to an identity, proven by some authorities. We call them certificate authorities, CAs, as we will see this term quite a lot.

Note that the certificate is not limited to TLS. In fact certificate is also used in another critical subsystem, identity of any players and components in Hyperledger Fabric. This is also the reason people get confused by these TLS with the identity in Fabric.

Now first take a look on a sample TLS server certificate. Inside a certificate we always find some essential information, and for our interests, among them, there are

  • subject’s identity
  • subject’s public key
  • issuer’s (CA’s) identity
  • issuer’s (CA’s) digital signature of this certificate
A sample certificate of a peer (peer0)

When presented with this certificate, one needs to verify if it is a valid certificate. A simplified but essential verification process is to use CA’s public key to verify the signature. If positive, we trust the subject has this identity and public key.

The remaining question is how we know the CA’s public key, as it is not seen in this sample certificate. And more importantly, why we trust this CA. Here we need the CA certificate (yes, this is another certificate in the same format, but this time it is the CA certificate, not the subject’s). In CA certificate we can obtain CA’s public key. Here is the CA certificate issuing the sample certificate above.

The CA Certificate issuing the certificate for peer0 above

Here we omit the very detail about CA, like types of CA, chain of certificate, chain of trust, etc. Let’s just focus on what interests us: as far as we have the CA certificate which we trust, we can verify the identity of a subject.

Complete picture for TLS process

By replacing the public key with certificates, we now have the full picture. Here is the flow.

  1. Alice sends a hello to show her intent to communicate with Bob with TLS.
  2. Bob presents a certificate which includes Bob’s identity and Bob’s public key, which is issued by a CA with CA’s signature.
  3. Alice trusts this CA, and verifies CA’s signature in Bob’s certificate. Upon successful, Alice knows she is talking to Bob, and inside the certificate Alice obtains Bob’s public key.
  4. Alice encrypts some secret using this public key.
  5. Bob obtains that secret by decrypting the received message with his own private key
  6. Based on some mathematics, Alice and Bob come up with some shared material, which can only be known by Alice and Bob, and cannot be deduced without Alice’s and Bob’s private key.
  7. With the same shared secret, now Alice and Bob can communicate safely their messages.

Observation

The following points are worthy of note :

This flow described above is exactly our daily experience when accessing a web page with https. When we access a TLS-enabled web site, we go through the same process. Just replace Alice with you (and your browser), and Bob with a TLS-enabled web server. Your browser by default is configured with a list of CA certificates that the browser trusts. If the server certificate received is signed by one CA certificate in this list, your browser accepts it automatically. That’s why most of the time we do not notice its existence.

In the process above only Bob is presenting a certificate to Alice. We say Alice authenticates Bob (to make sure Bob is true Bob). This is one-side authentication. When needed, Bob can also authenticate Alice by requesting Alice’s certificate, in order to let Bob know he is also talking to true Alice. This is called mutual authentication. We will see this in Fabric.

Since the birth of SSL and TLS, the majority of usage is on client-server model. It is not a must: TLS can be used between two server entities. Nevertheless, terminology like client / server is still used in Fabric. The one-side authentication is always called server authentication. And the mutual authentication in fact is composed of client authentication and server authentication. It is a practice that we implement server authentication in TLS (like https), while client authentication is always an option. Again we will see this in Fabric.

TLS in Hyperledger Fabric

Overview

With some basic ideas about TLS, we can now see how TLS is used in Fabric. TLS is securing communication between any two parties inside a fabric network. These parties can be a node such as peer and orderer, a Certificate Authority server which can be used for identity and/or TLS, client application invoking or querying chaincode functions, etc.

As mentioned in the beginning, some may find a confusion between TLS in Fabric with the identity subsystem in a fabric network, as both are implemented with certificate authorities. As a matter of fact, the identity part, or known as membership service provider (MSP), is a far more critical element in a fabric network as a permissioned blockchain environment. TLS is technically an option that can be even turned off without impacting the fabric operation.

In a fabric network there is no direct relationship between the TLS and identity. We can safely consider them as two separate parts inside a fabric network.

TLS Options in a Fabric Network

There are three possible options when using TLS in a fabric network.

Server Authentication

This is considered the most common use of TLS in a fabric network. This is the default setting according to readthedocs, which is being used in many sample networks. Server authentication happens when a client application accesses a peer or an orderer (e.g. invoking or querying chaincode functions). It can also be used when a Fabric CA Client accessing a Fabric CA Server. We can simply relate this use to how we access a TLS-enabled web server with our browser.

In this case, when a client requests TLS communication to a network component (peer, orderer or Fabric CA server), it presents its TLS server certificate to the client. The client (client application, Fabric CA Client) uses a given CA certificate to verify the server certificate. Upon success, a TLS communication is established as described in the TLS part, and the client proceeds with other fabric operations.

Mutual Authentication (Client Authentication enabled)

Both sides require authentication to each other. This can simply be considered as the addition of client authentication to the server authentication mentioned above: network component also requests client to present client’s certificate for authentication. TLS communication is established only after both authentications are successful. Otherwise we will see communication errors.

TLS Disabled

It is always technically possible to have TLS disabled. In this case, both parties are not authenticated to each other, and communication between parties is in clear text. It is not recommended for production, and is usually used for testing or learning.

The choice of which option is purely due to security policy for the consortium. It does not have impacts on other fabric operations.

Observing a Typical TLS Operation

Here we use chaincode function invocation to show how TLS is used. After deploying the Test Network and chaincode sacc, we invoke the chaincode function set. There are several TLS CA implementations, and in this example, for each organization, one TLS CA is created.

cd fabric-samples/test-network
./network.sh up createChannel
./network.sh deployCC -ccn mycc -ccp ../chaincode/sacc/

As a quick note, chaincode invocation involves both the endorser(s) to simulate the chaincode execution, and orderer for transaction processing. As server authentication is enabled, in the peer command we need to specify the CA certificate for each node.

peer chaincode invoke -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com --tls true --cafile ${PWD}/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n mycc --peerAddresses localhost:7051 --tlsRootCertFiles ${PWD}/organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses localhost:9051 --tlsRootCertFiles ${PWD}/organizations/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -c '{"Args":["set","name","Peter"]}'peer chaincode query -C mychannel -n mycc -c '{"Args":["get","name"]}'

Here we notice three TLS CA certificates specified.

  • For ordering service, target is localhost:7050 (orderer.example.com), and the TLS CA file is specified with cafile.
  • For endorsement, targets are localhost:7051 (peer0.org1.example.com) and localhost:9051 (peer0.org2.example.com). The two TLS CA files for these peers are specified with tlsRootCertFiles.

This is how we specify TLS CAs for different nodes with peer command. When a node (orderer or peer) presents its own server certificate, the client uses the corresponding TLS CA certificate to verify the server certificate.

Let us simulate situations when proper TLS CA is not specified.

Remove the TLS CA for orderer (no cafile specified). We see “failed to create connection” as TLS is enabled but Orderer’s CA certificate is not provided.

peer chaincode invoke -o localhost:7050 -C mychannel -n mycc --peerAddresses localhost:7051 --tlsRootCertFiles ${PWD}/organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses localhost:9051 --tlsRootCertFiles ${PWD}/organizations/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -c '{"Args":["set","name","John"]}'
no orderer TLS CA specified with cafile option

Or we specify a wrong certificate (use TLS CA for Org1 in peer0.org2.example.com). We will see the connection to peer0.org2 (localhost:9051) fails for the same reason: failed to create new connection.

peer chaincode invoke -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com --tls true --cafile ${PWD}/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n mycc --peerAddresses localhost:7051 --tlsRootCertFiles ${PWD}/organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses localhost:9051 --tlsRootCertFiles ${PWD}/organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt -c '{"Args":["set","name","John"]}'
Wrong CA certificate for localhost:9051 (peer0.org2.example.com)

Configuration of TLS options

With the basic operation on TLS using peer command, we can examine the configuration of TLS for these three options, and make some observations on them.

Server Authentication (the default)

To turn on server authentication for any network components, first make sure proper crypto material (private key and certificate) is generated. It can be done with the cryptogen, Fabric CA Server, or any other third party tools, as far as proper directory structure is met, and path of crypto material is properly specified in the configuration.

Here we take a look on the Test Network. We will see the docker-compose configuration for peer0.org1.example.com. It is an implementation of server authentication.

peer0.org1.example.com in docker/docker-compose-test-net.yaml

We see several variables are set for this peer (container).

  • TLS server authentication is enabled.
  • TLS crypto material, including server private key, server certificate, and the certificate of CA which issues the server certificate.
  • Volume mapped from the localhost to the path of crypto material inside the container.

With these variables, when peer node start is executed, the peer is ready for communicating with TLS in server authentication.

Mutual Authentication (Client Authentication enabled)

We will use Test Network with modification to show how client authentication works. We include the peer0.org1.example.com with this environment variable CORE_PEER_TLS_CLIENTAUTHREQUIRED to true (by default it is false).

Variable to turn on requiring client authentication for this peer

Here we show the demonstration.

Bring up the network. This involves generation of crypto material and bringing up the containers according to the docker-compose file. No peer command is issued yet.

./network.sh up

Now use the script to bring up a channel. Note that in the script peer command does not have client authentication when accessing peer0.org1.example.com. As a result, the script stops after several attempts.

./network.sh createChannel
Script fails after several attempts joining peer0.org1.example.com to mychannel

The original command in the script is like this. By default the client authentication is not enabled. Therefore the peer channel join command fails.

peer channel join -b channel-artifacts/mychannel.block

To specify client authentication in this peer command, we need to specify the variable in this peer command. Also note that a client certificate is generated in cryptogen for users.

CORE_PEER_TLS_CLIENTAUTHREQUIRED=true CORE_PEER_TLS_CLIENTCERT_FILE=/home/ubuntu/fabric-samples/kc-test-network/organizations/peerOrganizations/org1.example.com/users/[email protected]/tls/client.crt CORE_PEER_TLS_CLIENTKEY_FILE=/home/ubuntu/fabric-samples/kc-test-network/organizations/peerOrganizations/org1.example.com/users/[email protected]/tls/client.key peer channel join -b channel-artifacts/mychannel.block

With this, the peer channel join is successful.

Successfully joining peer0.org1.example.com to mychannel with client authentication information

As a comparison, there is no client authentication requirement in peer0.org2.example.com, and we can use the peer command directly to join peer0.org2.example.com.

Peer0.org2.example.com does not require client authentication

This is how we turn on client authentication on peer. Similar setup for orderer with different environment variables. See this for detail about variables for orderers.

TLS Disabled

We can turn off TLS in docker-compose file. Here again we turn off peer0.org1.example.com by setting CORE_PEER_TLS_ENABLED to false. Don’t forget to remove the client authentication variable added in the previous session.

Variable to turn on/off TLS for this peer

We perform a similar demonstration as previous part, and we encounter the same failure when we try to join the peer0.org1.example.com to mychannel.

The reason is that my terminal has this environment variable CORE_PEER_TLS_ENABLEDset to true.

my terminal has turned on TLS.

To override this, we specify it false.

CORE_PEER_TLS_ENABLED=false peer channel join -b channel-artifacts/mychannel.block
Successfully joining peer0.org1.example.com to mychannel with TLS turned off

Observation

No matter whether TLS is turned on or not, and no matter whether client authentication is used or not, fabric operation can still work well. Fabric messages can be sent whether there is TLS or not, and whether client authentication is used or not. There is no direct dependency between TLS and general fabric operation.

The selection of TLS configuration options is made at individual node level. In real life it is desired to have a single security policy applied to all nodes in a network (e.g. all communication between components must be TLS enabled).

TLS CA Implementation

The final puzzle of the TLS is how to implement TLS CA in a fabric network. There is no single must-use setup, as the implementation depends largely on the business needs and hence the fabric network design. Therefore it always helps to observe some working models and make reference to them.

So far we have seen three implementations in the readthedocs. In this session we will take a look on all of the three.

Each organization has two CAs: one for TLS, one for identity (MSP)

This is the setup for Test Network when cryptogen is used for crypto material generation. In this case, two CAs are created for each organization: one for TLS and one for identity (MSP).

Each organization has two CAs, one for TLS and one for identity (MSP)

Let’s make some observations.

./network.sh up

We are inspecting the certificates from peer0.org1.example.com and peer0.org2.example.com, both TLS and identity (MSP). We see two different CAs in each organization.

For peer0.org1.example.com (different issuer for TLS and identity)

For peer0.org2.example.com (different issuer for TLS and identity, and both different from Org1)

Each organization has one CA, serving both TLS and identity

As mentioned before, although there is no dependency between identity CA and TLS CA for any organization, there is also no restriction that we cannot use one single CA for both purposes.

This is the setup for Test Network when CA is used for crypto material generation. In this case, one CA is created in each organization.

Each organization has one CA, serving both TLS and identity (MSP)

Let’s make some observations.

./network.sh up -ca

We are inspecting the certificates from peer0.org1.example.com and peer0.org2.example.com, both TLS and identity (MSP). We see two different CAs in each organization.

For peer0.org1.example.com (same issuer for both TLS and identity)

For peer0.org2.example.com (same issuer for both TLS and identity, different from org1)

One TLS CA serving all organizations

This is a setup in Fabric CA Operations Guide tutorial (link). In this setup, one TLS CA issues TLS certificates for all organizations.

We have developed the scripts performing this deployment. For detail please see this article.

One TLS CA for all organizations, while each organization has its own identity CA.

Here we just inspect the result.

For peer1-org1 (different CA for TLS and identity)

For peer1-org2 (same TLS CA as peer1-org1, different identity CA)

We can conclude that there can be many different setups on TLS-CA, and these setups are all working well. In our design, choose one of the three deployments as a start, and make proper modification to fit actual business needs.

Summary

Through the discussion and demonstration, we can see that

  • Hyperledger Fabric uses TLS to secure communication between any components inside a network.
  • TLS relies on a public key infrastructure (certificate authorities), and so does the identity (MSP) in a fabric network. We can treat them into two different and independent parts.
  • Unlike the MSP, TLS in Hyperledger Fabric is an option. The fabric system can run well without TLS, although in production TLS is always preferred.
  • When TLS is used, we can decide whether server authentication or mutual authentication is needed according to the security policy of the consortium.
  • There are different ways to implement TLS CA, and the choice is again to meet the security policy of the consortium.

Hope you have gained some understanding on TLS in Fabric.