Aeraki Mesh – Blog

Blog: Database Mesh: Simplify Redis Cluster with Istio and Aeraki

Thu, 11 May 2023 00:00:00 +0000

Redis is a high-performing key-value database known for its versatility in supporting a diverse range of data structures and operations, including strings, hashes, lists, sets, and sorted sets. Its robust capabilities make it an ideal choice for caching, session storage, message brokers, and other similar applications.

Istio is one of the most popular service mesh platforms that provides a unified way to connect, secure, and manage microservices. Aeraki Mesh is a CNCF open-source project that works with Istio and enhances Istio’s capabilities by providing advanced traffic management for non-HTTP protocols including Thrift, Dubbo, Redis, and proprietary protocols.

Managing a Redis Cluster can be complicated. Istio and Aeraki can help with that. This post uses a demo to demonstrate how to use Istio and Aeraki to manage Redis traffic for your applications, providing client-transparent advanced features such as data sharding, prefix routing, read/write separation, traffic mirroring, fault injection, etc.

System architecture

Aeraki and Istio operate on the control plane, while the data plane is composed of Envoy sidecars. In the control plane, Aeraki provides two user-friendly Kubernetes Custom Resource Definitions (CRDs), RedisService and RedisDestination, as the rules for users to manage Redis traffic. Aeraki converts these traffic rules into data plane configurations and deploys them to Envoy sidecars through xDS. Envoy sidecars work on the data plane, intercepting Redis client requests and providing relevant traffic management capabilities based on the configuration deployed from the control plane.

Install the demo

First, download Aeraki Mesh from GitHub.

$ git clone https://github.com/aeraki-mesh/aeraki.git

Before installing Aeraki, it is necessary to first install Istio since Aeraki is built on top of Istio. However, the Aeraki installation script takes care of Istio installation automatically, so executing the Aeraki installation script is all that is required.

$ make demo

Execute the following script. This script creates a namespace called redis and deploy the Redis demo application within it.

$ ./demo/redis/install.sh

The demo application includes a 6-node Redis cluster (a Kubernetes StatefulSet), a single-mode Redis instance, and a Redis client. Later on, this demo will be used to demonstrate Aeraki Mesh’s traffic management capabilities for Redis.

$ kubectl -n redis get pod
NAME READY STATUS RESTARTS AGE
redis-client-644c965f48-dvjc7 2/2 Running 0 2m17s
redis-cluster-0 1/1 Running 0 2m17s
redis-cluster-1 1/1 Running 0 2m15s
redis-cluster-2 1/1 Running 0 2m13s
redis-cluster-3 1/1 Running 0 2m12s
redis-cluster-4 1/1 Running 0 2m11s
redis-cluster-5 1/1 Running 0 2m10s
redis-single-ccbb984dc-qz22v 1/1 Running 0 2m17s

Redis server authentication

In the demo environment, the redis-single service has been configured with an access password. Attempting to access this service via a client will result in Redis prompting the user for authentication and denying the request.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-single
redis-single:6379> set a a
(error) NOAUTH Authentication required.

Without Aeraki Mesh, the client needs to know whether authentication is required, as well as any necessary usernames and passwords for authentication. This adds complexity to client code.

By using Aeraki Mesh’s RedisDestination, the password for accessing Redis can be set at the runtime, and authentication between the client and Redis server can be handled by the Sidecar Proxy. This eliminates the need for the Client to manage password information for the Redis service.

$ kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: redis-service-secret
 namespace: redis
type: Opaque
data:
 password: dGVzdHJlZGlzCg==
---
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisDestination
metadata:
 name: redis-single
 namespace: redis
spec:
 host: redis-single.redis.svc.cluster.local
 trafficPolicy:
 connectionPool:
 redis:
 auth:
 secret:
 name: redis-service-secret
EOF

With this Aeraki Redis traffic rule, you should be able to access the redis-single service without any issues.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-single
redis-single:6379> set foo bar
OK

Client-side authentication

In some cases, you may want the access password used by the client to differ from the actual password for the Redis service. Doing so provides a number of advantages, including the ability to change the Redis service password without impacting the client and reducing the likelihood of exposing sensitive credentials. Through RedisService, clients can be assigned separate access passwords to meet these needs.

$ kubectl apply -f- <<EOF
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisService
metadata:
 name: redis-single
 namespace: redis
spec:
 host:
 - redis-single.redis.svc.cluster.local
 settings:
 auth:
 plain:
 password: testredis123!
EOF

To access the redis-single service at this time, the new password set within the above RedisService must be used.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-single
redis-single:6379> AUTH testredis
(error) ERR invalid password
redis-single:6379> AUTH testredis123!
OK
redis-single:6379> get foo
"bar"

Note: Both the RedisDestination and RedisService “auth” fields support two different methods for obtaining keys: “secret” and “plain”. “Secret” indicates that any necessary username and password information can be found inside a Kubernetes secret resource. In such cases, the default username value for “auth” will be the “username” defined within the specified secret, with either the “password” or “token” serving as the associated password for “auth”. However, if alternative fields within the secret are used for authentication purposes, the configuration of “passwordField” and “usernameField” can be used to specify which password and username fields should be utilized for authentication.

Making Redis deployment mode transparent to clients

Redis can be deployed in either single-node or cluster mode. Cluster mode offers several advantages over single-node mode, including:

Supports horizontal scaling by increasing the number of nodes to improve performance and capacity.
Supports automatic partitioning to distribute data across different nodes, enhancing load balancing and availability.
Provides a certain degree of fault tolerance, enabling automatic failover and recovery even if a node fails or network issues arise.

Redis requires different client APIs for accessing single-node and cluster modes. However, by leveraging Aeraki Mesh’s Redis traffic management features, you can easily switch between these two modes without needing to modify client code, thus simplifying application development.

As an example, you might use a small, single-instance Redis service for testing purposes, while deploying a Redis cluster consisting of multiple instances for production workloads. Through the use of Aeraki Mesh, you can seamlessly connect our application to these different Redis services without needing to adjust application code or configurations, ensuring that development, staging, and production environments remain consistent throughout the software deployment cycle.

The Redis cluster deployed in the demo consists of six instances divided into three shards, each with one master node and one slave node. You can view the topology of this Redis cluster using the following command:

$ kubectl exec -it redis-cluster-0 -c redis -n redis -- redis-cli cluster shards

The Redis cluster’s topology is depicted in the above diagram, which shows that the cluster consists of three shards, with each shard responsible for a specific range of slots. Shard 0 handles slots 0 through 5460, Shard 1 handles slots 5461 through 10922, and Shard 2 handles slots 10923 through 16383. Importantly, each key is associated with a specific slot number, which is calculated using the formula CRC16(key) mod 16384.

Therefore, when writing or reading data to the Redis cluster, clients must first calculate the slot number based on the CRC16(key) mod 16384 algorithm, and then send the request to the corresponding node in the appropriate shard.

If you attempt to access the Redis cluster in the demo, you may encounter the following access error:

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> set foo bar
(error) MOVED 15495 10.244.0.23:6379

The reason for this error is that the Redis client is using a regular API, which means that requests are sent to the redis-cluster service and then forwarded to a random Redis node in the cluster. If that node is not responsible for handling the corresponding slot of the key in the request, the node will return a MOVED error indicating which node the client should connect to instead.

Unfortunately, since the client is using a regular API, it is unable to interpret this error and automatically resend the request to the appropriate node. As a result, the client continues to return an error.

By setting the mode parameter in RdisDestination to CLUSTER, Aeraki Mesh is able to abstract away the differences between Redis’ Cluster mode and standalone mode.

$ kubectl apply -f- <<EOF
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisDestination
metadata:
 name: external-redis
 namespace: redis
spec:
 host: external-redis.redis.svc.cluster.local
 trafficPolicy:
 connectionPool:
 redis:
 mode: CLUSTER
EOF

Now, you can interact with the redis-cluster service just as you would with a standalone Redis node.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> set foo bar
OK

With this approach, you can seamlessly switch between development and production environments without modifying application code. Additionally, as our business grows and the demands on our Redis infrastructure become too burdensome, you can effortlessly migrate from a single-node Redis deployment to a Cluster.

In a Redis cluster, different keys are stored across different shards, allowing us to scale up by either increasing the number of replica nodes within each shard or adding additional shards to the cluster itself. This ensures that you can effectively manage any increases in data pressure that may result from ongoing business expansion. By integrating with Envoy proxy and leveraging its powerful traffic management functionality, Aeraki Mesh makes the entire migration and scaling process fully transparent, ensuring that online business operations remain uninterrupted throughout.

Prefix routing

You can route traffic to different Redis services based on the prefix of the requested key with the help of RedisService.

As an example, consider the following RedisService, which routes any key beginning with “cluster” to the redis-cluster service, while all other keys are directed to the redis-single service.

$ kubectl apply -f- <<EOF
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisService
metadata:
 name: redis-cluster
 namespace: redis
spec:
 host:
 - redis-cluster.redis.svc.cluster.local
 redis:
 - match:
 key:
 prefix: cluster
 route:
 host: redis-cluster.redis.svc.cluster.local
 - route:
 host: redis-single.redis.svc.cluster.local
EOF

With this configuration in place, first, you use a Redis client to access the redis-cluster service and set values for two distinct keys, “test-route” and “cluster-test-route”. You can then retrieve the values of these keys using the get command.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> set test-route "this key goes to redis-single"
OK
redis-cluster:6379> set cluster-test-route "this key goes to redis-cluster"
OK
redis-cluster:6379> get test-route
"this key goes to redis-single"
redis-cluster:6379> get cluster-test-route
"this key goes to redis-cluster

If you connect to the redis-single service, you can see that it only contains the “test-route” key, while the value of the “cluster-test-route” key is nil. This behavior confirms our RedisService configuration, as it shows that “test-route” has been correctly routed to the redis-single service, while “cluster-test-route” is being handled by the redis-cluster service.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-single
redis-single:6379> AUTH testredis123!
OK
redis-single:6379> get test-route
"this key goes to redis-single"
redis-single:6379> get cluster-test-route
(nil)

Read/write separation

In Redis Cluster, there are multiple shards, each consisting of a master node and one or more replica(slave) nodes. The master nodes are responsible for handling write operations and synchronizing updates with their associated replicas. Replicas, in turn, serve as backup nodes and can respond to client read requests since they maintain an identical dataset to that of their master.

Aeraki Mesh supports setting different read policies for Redis through RedisService:

MASTER: the default read mode. It only reads data from the master node and should be used when strong consistency is required. This mode places significant pressure on the master node and cannot distribute read workload among replicas within the same shard.
PREFER_MASTER: prioritizes reading data from the master node, but will fall back to replica nodes if it becomes unavailable.
REPLICA: only read data from the replica nodes. Since the replication process between master and replica is asynchronous, this mode may return outdated data and is therefore suitable for scenarios where clients do not require strict data consistency. Multiple replicas can effectively distribute read load in this mode.
PREFER_REPLICA: prioritize reading data from the replica nodes, but will revert to the master node if they become unavailable.
ANY: read data from any available node.

By setting the read mode to REPLICA, you can reduce the workload on the master node by having it only handle write operations while the replica nodes handle read operations. As our business grows, you can also increase the number of replica nodes within a shard to distribute read operations across multiple nodes.

$ kubectl apply -f- <<EOF
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisService
metadata:
 name: redis-cluster
 namespace: redis
spec:
 host:
 - redis-cluster.redis.svc.cluster.local
 settings:
 readPolicy: REPLICA 
 redis:
 - route:
 host: redis-cluster.redis.svc.cluster.local
EOF

Traffic mirroring

Using RedisService, you have the ability to duplicate any requests sent to a Redis service and route them to another Redis service concurrently. The client will only receive the response from the primary Redis service, while the response from the mirrored Redis service is discarded. You can also set the percentage of traffic that is mirrored, for example, mirroring 50% of the traffic to another Redis service.

For example, consider the following RedisService configuration, which mirrors any requests sent to the redis-cluster service to the redis-single service:

$ kubectl apply -f- <<EOF
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisService
metadata:
 name: redis-cluster
 namespace: redis
spec:
 host:
 - redis-cluster.redis.svc.cluster.local
 redis:
 - route:
 host: redis-cluster.redis.svc.cluster.local
 mirror:
 - route:
 host: redis-single.redis.svc.cluster.local
 percentage:
 value: 100
EOF

Apply the above configuration, then let’s connect to redis-cluster and set the value of test-traffic-mirroring.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> set test-traffic-mirroring "this key goes to both redis-cluster and redis-single"
OK
redis-cluster:6379> get test-traffic-mirroring
"this key goes to both redis-cluster and redis-single"

Now if you connect to redis-single, you can see that the key “test-traffic-mirroring” also exists in the redis-single service. This implies that the requests sent to the redis-cluster were mirrored to the redis-single.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-single
redis-single:6379> AUTH testredis123!
OK
redis-single:6379> get test-traffic-mirroring
"this key goes to both redis-cluster and redis-single"

Fault injection

Using RedisService, you can inject faults into Redis services, which can be useful for conducting chaos testing and other scenarios to ensure that the system properly handles Redis failures.

RedisService supports two types of fault configurations:

Delay (DELAY): Adds delay to responses.
Error (ERROR): Returns errors on requests.

For instance, the following configuration will result in 50% of GET commands directly returning errors.

$ kubectl apply -f- <<EOF
apiVersion: redis.aeraki.io/v1alpha1
kind: RedisService
metadata:
 name: redis-cluster
 namespace: redis
spec:
 host:
 - redis-cluster.redis.svc.cluster.local
 redis:
 - route:
 host: redis-cluster.redis.svc.cluster.local
 faults:
 - type: ERROR
 percentage:
 value: 50
 commands:
 - GET
EOF

Apply the above configuration, then connect to redis-cluster through the client, half of the GET commands will return the error message Fault Injected: Error.

$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> get a
(error) Fault Injected: Error
redis-cluster:6379> get a
"a"

Connect to external redis

While the Redis service in the demo is deployed within a Kubernetes cluster, it’s possible to use Aeraki Mesh to connect to a Redis service that’s outside of the cluster. This can be done by creating a service without selectors, followed by creating an EndpointSlice for the service to specify the address of the external Redis. Once that’s done, RedisService and Redis Destination can be used to manage traffic for the service, just as they would for Redis within the cluster.

$ kubectl apply -f- <<EOF 
apiVersion: v1
kind: Service
metadata:
 name: external-redis
 namespace: redis
spec:
 ports:
 - name: tcp-redis
 protocol: TCP
 port: 6379
 targetPort: 6379
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
 name: external-redis
 namespace: redis
 labels:
 kubernetes.io/service-name: external-redis
addressType: IPv4
ports:
 - name: tcp-redis
 port: 6379
 protocol: TCP
endpoints:
 - addresses:
 - 10.244.0.26 # 集群外 Redis 实例的地址，比如云厂商提供的 Redis 服务。
EOF

Wrapping up

This article has demonstrated how to use Aeraki Mesh for traffic management on Redis, covering features such as authentication, redis cluster, prefix routing, traffic mirroring, and fault injection. These functions are now available in the latest version of Aeraki Mesh, and we encourage everyone to try them out and provide feedback and suggestions. For more information about Aeraki Mesh, please refer to its official website and GitHub repository.

References

Aeraki Mesh: https://aeraki.net/
GitHub: https://github.com/aeraki-mesh

Blog: Aeraki Mesh Community Meeting - February 23rd, 2023

Fri, 24 Feb 2023 00:00:00 +0000

During this community meeting, we introduced Aeraki Mesh’s newly released Access Log and Service Metrics features. We also invited experts from Boss Zhipin and Shopline to share hands-on examples of Dubbo based on Aeraki Mesh. Furthermore, we introduced the new features of multiplexing and Gateway support for the MetaProtocol application protocol, which are being contributed by the Tencent Games project and Boss Direct.

The Access Log and Service Metrics functionalities will help users better understand their network performance, troubleshoot issues, and optimize their applications. We’re excited to see how these new features will improve our users' experiences with Aeraki Mesh.

During the meeting, the experts from Boss Zhipin and Shopline demonstrated how Dubbo, a popular open-source RPC framework, can be used with Aeraki Mesh. Their presentations provided valuable insights and hands-on examples for users who are interested in using Dubbo with Aeraki Mesh.

We are also thrilled to announce the addition of multiplexing for the Aeraki Mesh data plane - MetaProtocol Proxy and Gateway support for non-HTTP protocols. These features will enhance the functionality of the Tencent Games project and Boss Zhipin, and we believe that our users will benefit greatly from their contributions.

Thank you to everyone who attended the community meeting. We look forward to continuing to improve Aeraki Mesh and providing our users with the best possible experience.

Slides

Videos

Bilibili

YouTube

Blog: Istiocon 2022 Talk: Tencent Music’s service mesh practice with Istio and Aeraki

Tue, 26 Apr 2022 00:00:00 +0000

This session talked about Tencent music’s service mesh practice with Istio and Aeraki. Including:

How to extend Istio with Aeraki to manage the traffic of proprietary protocols
Deep dive into Aeraki and MetaProtcol Proxy
How Tencent Music leverage Istio and Aeraki to build a fully functional service mesh, managing both the HTTP and proprietary protocols

Slides

Download pdf

Recording

Bilibili

YouTube

Blog: How Aeraki Mesh are Used for Game Streaming During the 2020 Winder Olympics

Wed, 30 Mar 2022 00:00:00 +0000

On this live broadcast, My colleague and I presented a production use case of Aeraki Mesh with the Yangshiping App, which has been used for streaming games during the 2022 Winter Olympics.

What’s in the presentation?

Introduce the second version of Aeraki Mesh architecture, features, and future roadmap.
Aeraki Mesh layer-7 proxy framework MetaProtocol deep dive.
How to implement a private protocol on top of the MetaProtocol proxy, and manage its traffic within Aeraki Mesh.
Aeraki Mesh use case study: video streaming protocol: videopacket.
Aeraki Mesh use case study: HPA based on rate limiting metrics.
Aeraki Mesh use case study: Traffic mirroing and advanced routing.

Slides deck

Dowload slides

Recording

Bilibili

YouTube

Blog: Aeraki Roadmap 2022

Thu, 10 Feb 2022 00:00:00 +0000

Blog: Aeraki — Manage Any Layer-7 Protocol in Istio Service Mesh

Tue, 28 Sep 2021 00:00:00 +0000

Aeraki [Air-rah-ki] is the Greek word for ‘breeze’. While Istio connects microservices in a service mesh, Aeraki provides a framework to allow Istio to support more layer-7 protocols other than just HTTP and gRPC. We hope this breeze can help Istio sail a little further.

Lack of Protocols Support in Service Mesh

We are now facing some challenges with service meshes:

Istio and other popular service mesh implementations have very limited support for layer 7 protocols other than HTTP and gRPC.
Envoy RDS(Route Discovery Service) is solely designed for HTTP. Other protocols such as Dubbo and Thrift can only use listener in-line routes for traffic management, which breaks existing connections when routes change.
It takes a lot of effort to introduce a proprietary protocol into a service mesh. You’ll need to write an Envoy filter to handle the traffic in the data plane, and a control plane to manage those Envoys.

Those obstacles make it very hard, if not impossible, for users to manage the traffic of other widely-used layer-7 protocols in microservices. For example, in a microservices application, we may have the below protocols:

RPC: HTTP, gRPC, Thrift, Dubbo, Proprietary RPC Protocol …
Messaging: Kafka, RabbitMQ …
Cache: Redis, Memcached …
Database: MySQL, PostgreSQL, MongoDB …

If you have already invested a lot of effort in migrating to a service mesh, of course, you want to get the most out of it — managing the traffic of all the protocols in your microservices.

Aeraki’s approach

To address these problems, we create an open-source project, Aeraki, to provide a non-intrusive, extendable way to manage any layer-7 traffic in an Istio service mesh.

As this diagram shows, Aeraki Mesh consists of the following components:

Aeraki: Aeraki provides high-level, user-friendly traffic management rules to operations, translates the rules to envoy filter configurations, and leverages Istio’s EnvoyFilter API to push the configurations to the sidecar proxies. Aeraki also serves as the RDS server for MetaProtocol proxies in the data plane. Contrary to Envoy RDS, which focuses on HTTP, Aeraki RDS is aimed to provide a general dynamic route capability for all layer-7 protocols.
MetaProtocol Proxy: MetaProtocol Proxy provides common capabilities for Layer-7 protocols, such as load balancing, circuit breaker, load balancing, routing, rate limiting, fault injection, and auth. Layer-7 protocols can be built on top of MetaProtocol. To add a new protocol into the service mesh, the only thing you need to do is implementing the codec interface and a couple of lines of configuration. If you have special requirements which can’t be accommodated by the built-in capabilities, MetaProtocol Proxy also has an application-level filter chain mechanism, allowing users to write their own layer-7 filters to add custom logic into MetaProtocol Proxy.

Dubbo and Thrift have already been implemented based on MetaProtocol. More protocols are on the way. If you’re using a close-source, proprietary protocol, you can also manage it in your service mesh simply by writing a MetaProtocol codec for it.

Most request/response style, stateless protocols can be built on top of the MetaProtocol Proxy. However, some protocols' routing policies are too “special” to be normalized in MetaProtocol. For example, Redis proxy uses a slot number to map a client query to a specific Redis server node, and the slot number is computed by the key in the request. Aeraki can still manage those protocols as long as there’s an available Envoy Filter in the Envoy proxy side. Currently, for protocols in this category, Redis and Kafka are supported in Aeraki.

Deep Dive Into MetaProtocol

Let’s look into how MetaProtocol works. Before MetaProtocol is introduced, if we want to proxy traffic for a specific protocol, we need to write an Envoy filter that understands that protocol and add the code to manipulate the traffic, including routing, header modification, fault injection, traffic mirroring, etc.

For most request/response style protocols, the code for traffic manipulation is very similar. Therefore, to avoid duplicating these functionalities in different Envoy filters, Aeraki Mesh implements most of the common functions of a layer-7 protocol proxy in a single place — the MetaProtocol Proxy filter.

This approach significantly lowers the barrier to write a new Envoy filter: instead of writing a fully functional filter, now you only need to implement the codec interface. In addition to that, the control plane is already in place — Aeraki works at the control plane to provides MetaProtocol configuration and dynamic routes for all protocols built on top of MetaProtocol.

There are two important data structures in MetaProtocol Proxy: Metadata and Mutation. Metadata is used for routing, and Mutation is used for header manipulation.

At the request path, the decoder(the decode method of the codec implementation) populates the Metadata data structure with key-value pairs parsed from the request, then the Metadata will be passed to the MetaProtocol Router. The Router selects an appropriate upstream cluster after matching the route configuration it receives from Aeraki via RDS and the Metadata.

A custom filter can populate the Mutation data structure with arbitrary key-value pairs if the request needs to be modified: adding a header or changing the value of a header. Then the Mutation data structure will be passed to the encoder(the encode method of the codec implementation). The encoder is responsible for writing the key-value pairs into the wire protocol.

The response path is similar to the request path, only in a different direction.

An Example

If you need to implement an application protocol based on MetaProtocol, you can follow the below steps(use Thrift as an example):

Data Plane

Implement the codec interface to encode and decode the protocol package. You can refer to Dubbo codec and Thrift codec as writing your own implementation.
Define the protocol with Aeraki ApplicationProtocol CRD, as this YAML snippet shows:

apiVersion: metaprotocol.aeraki.io/v1alpha1
kind: ApplicationProtocol
metadata:
 name: thrift
 namespace: istio-system
spec:
 protocol: thrift
 codec: aeraki.meta_protocol.codec.thrift

Control Plane

You don’t need to implement the control plane. Aeraki watches services and traffic rules, generates the configurations for the sidecar proxies, and sends the configurations to the data plane via EnvoyFilter and MetaProtocol RDS.

Protocol selection

Similar to Istio, protocols are identified by service port prefix. Please name service ports with this pattern: tcp-metaprotocol-{application protocol}-xxx. For example, a Thrift service port should be named tcp-metaprotocol-thrift.

Traffic management

You can change the route via MataRouter CRD. For example: send 20% of the requests to v1 and 80% to v2:

apiVersion: metaprotocol.aeraki.io/v1alpha1
kind: MetaRouter
metadata:
 name: test-metaprotocol-route
spec:
 hosts:
 - thrift-sample-server.thrift.svc.cluster.local
 routes:
 - name: traffic-spilt
 route:
 - destination:
 host: thrift-sample-server.thrift.svc.cluster.local
 subset: v1
 weight: 20
 - destination:
 host: thrift-sample-server.thrift.svc.cluster.local
 subset: v2
 weight: 80

Hope this helps if you need to manage protocols other than HTTP in a service mesh. Reach out to zhaohuabing if you have any questions.

Reference

Aeraki GitHub
Live Demo: Kiali Dashboard
Live Demo: Service Metrics: Grafana
Live Demo: Service Metrics: Prometheus
Istio meetup China(Chinese): Full Stack Service Mesh - Manage Any Layer-7 Traffic in an Istio Service Mesh with Aeraki
IstioCon 2021: How to Manage Any Layer-7 Traffic in an Istio Service Mesh?