diff --git a/sentinel-cluster/sentinel-cluster-server-envoy-rls/Dockerfile b/sentinel-cluster/sentinel-cluster-server-envoy-rls/Dockerfile new file mode 100644 index 00000000..7a1f2440 --- /dev/null +++ b/sentinel-cluster/sentinel-cluster-server-envoy-rls/Dockerfile @@ -0,0 +1,10 @@ +FROM java:8-jre + +ENV SENTINEL_HOME /sentinel + +COPY target/sentinel-envoy-rls-token-server.jar $SENTINEL_HOME/ + +WORKDIR $SENTINEL_HOME + +ENTRYPOINT ["sh", "-c"] +CMD ["java -Dcsp.sentinel.log.dir=/sentinel/logs/ ${JAVA_OPTS} -jar sentinel-envoy-rls-token-server.jar"] \ No newline at end of file diff --git a/sentinel-cluster/sentinel-cluster-server-envoy-rls/README.md b/sentinel-cluster/sentinel-cluster-server-envoy-rls/README.md index 88b337e0..e6100436 100644 --- a/sentinel-cluster/sentinel-cluster-server-envoy-rls/README.md +++ b/sentinel-cluster/sentinel-cluster-server-envoy-rls/README.md @@ -13,3 +13,47 @@ Build the executable jar: ```bash mvn clean package -P prod ``` + +## Rule configuration + +Currently Sentinel RLS token server supports dynamic rule configuration via the yaml file. +The file may provide rules for one *domain* (defined in Envoy's conf file). +In Envoy, one rate limit request might carry multiple *rate limit descriptors* +(which will be generated from [Envoy rate limit actions](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/route/route.proto#envoy-api-msg-route-ratelimit)). +One rate limit descriptor may have multiple entries (key-value pair). +We may set different threshold for each rate limit descriptors. + +A sample rule configuration file: + +```yaml +domain: foo +descriptors: + - resources: + - key: "destination_cluster" + value: "service_httpbin" + count: 1 +``` + +This rule only takes effect for domain `foo`. It will limit the max QPS to 1 for +all requests targeted to the `service_httpbin` cluster. + +We need to provide the path to yaml file via the `SENTINEL_RLS_RULE_FILE_PATH` env +(or `-Dcsp.sentinel.rls.rule.file` opts). Then as soon as the content in the rule file has been changed, +Sentinel will reload the new rules from the file to the `EnvoyRlsRuleManager`. + +We may check the logs in `~/logs/csp/sentinel-record.log.xxx` to see whether the rules has been loaded. +We may also retrieve the converted `FlowRule` via the command API `localhost:8719/cluster/server/flowRules`. + +## Configuration items + +The configuration list: + +| Item (env) | Item (JVM property) | Description | Default Value | Required | +|--------|--------|--------|--------|--------| +| `SENTINEL_RLS_GRPC_PORT` | `csp.sentinel.grpc.server.port` | The RLS gRPC server port | **10240** | false | +| `SENTINEL_RLS_RULE_FILE_PATH` | `csp.sentinel.rls.rule.file` | The path of the RLS rule yaml file | - | **true** | +| `SENTINEL_RLS_ACCESS_LOG` | - | Whether to enable the access log (`on` for enable) | off | false | + +## Samples + +- [Kubernetes sample](./sample/k8s) diff --git a/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/README.md b/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/README.md new file mode 100644 index 00000000..3fd58ba8 --- /dev/null +++ b/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/README.md @@ -0,0 +1,105 @@ +# Sentinel Envoy RLS - Kubernetes sample + +This sample will illustrate how to use Sentinel RLS token server with Envoy in Kubernetes clusters. + +## Build the Docker image + +We could use the pre-built Docker image: `registry.cn-hangzhou.aliyuncs.com/sentinel-docker-repo/sentinel-envoy-rls-server:latest` + +We can also manually build the Docker image in the `sentinel-cluster-server-envoy-rls` directory: + +```bash +docker build -t "sentinel/sentinel-envoy-rls-server:latest" -f ./Dockerfile . +``` + +## Deploy Sentinel RLS token server + +Next we could deploy the Sentinel RLS token server in the K8S cluster. +We've provided a deployment template for Sentinel RLS token server in `sentinel-rls.yml`. +It includes: + +- A `ConfigMap` that contains the cluster flow control rule for Envoy global rate limiting. + This will be mounted as a file in the target `Deployment`, so that the Sentinel RLS token server + could load the rules dynamically as soon as the rule in the `ConfigMap` has been updated. +- A `Deployment` for Sentinel RLS token server. It will mount the `ConfigMap` as a volume + for dynamic rule configuration. +- A `Service` that exports the Sentinel command port (8719) and the RLS gRPC port (by default 10245) + on a cluster-internal IP so that the Envoy pods could communicate with the RLS server. + +The sample rate limiting rule in the `sentinel-rule-cm`: + +```yaml +domain: foo +descriptors: + # For requests to the "service_httpbin" cluster, limit the max QPS to 1 + - resources: + - key: "destination_cluster" + value: "service_httpbin" + count: 1 +``` + +You may enable the access log in the Sentinel RLS token server (output to console) +via the `SENTINEL_RLS_ACCESS_LOG` env: + +```yaml +env: + - name: SENTINEL_RLS_ACCESS_LOG + value: "on" +``` + +You may also append JVM opts via the `JAVA_OPTS` env. + +After preparing the yaml template, you may deploy the Sentinel RLS token server: + +```bash +kubectl apply -f sample/k8s/sentinel-rls.yml +``` + +## Deploy Envoy + +Next we could deploy the Envoy instances in the K8S cluster. If you've already had Envoy instances running, +you could configure the address (`sentinel-rls-service`) and the port (`10245`) +of the rate limit cluster in your Envoy configuration. + +We've provided a deployment template for Envoy in `envoy.yml`. +It includes: + +- A `ConfigMap` that contains the configuration for Envoy. + This will be mounted as a file in the target `Deployment`, which will be loaded as the configuration + file by Envoy. +- A `Deployment` for Envoy. It will mount the `ConfigMap` as a volume + for configuration. +- A `Service` that exports the Envoy endpoint port (by default 10000) on a cluster-internal IP + so that it could be accessible from other pods. If you need external access, you could choose the + `LoadBalancer` type or add a frontend ingress. + +In the sample, we have two [Envoy clusters](https://www.envoyproxy.io/docs/envoy/latest/api-v2/clusters/clusters): + +- `service_httpbin`: HTTP proxy to `httpbin.org` +- `rate_limit_cluster`: the cluster of the Sentinel RLS token server + +This route configuration tells Envoy to route incoming requests to `httpbin.org`. In the `http_filters` conf item, +we added the `envoy.rate_limit` filter to the filter chain so that the global rate limiting is enabled. +We set the rate limit domain as `foo`, which matches the item in the Envoy RLS rule. +In the `route_config`, we provide the rate limit action: `{destination_cluster: {}}`, which will +generate the rate limit descriptor containing the actual target cluster name (e.g. `service_httpbin`). +Then we could set rate limit rules for each target clusters. + +After preparing the yaml template, you may deploy the Envoy instance: + +```bash +kubectl apply -f sample/k8s/envoy.yml +``` + +## Test the rate limiting + +Now it's show time! We could visit the URL `envoy-service:10000/json` in K8S pods. +Since we set the max QPS to 1, we could emit concurrent requests to the URL, and we +could see the first request passes, while the latter requests are blocked (status 429): + +![image](https://user-images.githubusercontent.com/9434884/68571798-d0a46500-049e-11ea-8488-5e90f56f23a5.png) + +## Update the rules dynamically + +You could update the rules in the `sentinel-rule-cm` ConfigMap. Once the content is updated, +Sentinel will perceive the changes and load the new rules to `EnvoyRlsRuleManager`. diff --git a/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/envoy.yml b/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/envoy.yml new file mode 100644 index 00000000..b7326158 --- /dev/null +++ b/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/envoy.yml @@ -0,0 +1,132 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: envoy-cm +data: + envoy-yml: |- + admin: + access_log_path: /tmp/admin_access.log + address: + socket_address: + protocol: TCP + address: 127.0.0.1 + port_value: 9901 + static_resources: + listeners: + - name: listener_0 + address: + socket_address: + protocol: TCP + address: 0.0.0.0 + port_value: 10000 + filter_chains: + - filters: + - name: envoy.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager + stat_prefix: ingress_http + route_config: + name: local_route + virtual_hosts: + - name: local_service + domains: ["*"] + routes: + - match: + prefix: "/" + route: + host_rewrite: httpbin.org + cluster: service_httpbin + rate_limits: + - stage: 0 + actions: + - {destination_cluster: {}} + http_filters: + - name: envoy.rate_limit + config: + domain: foo + stage: 0 + rate_limit_service: + grpc_service: + envoy_grpc: + cluster_name: rate_limit_cluster + timeout: 0.25s + - name: envoy.router + clusters: + - name: service_httpbin + connect_timeout: 0.5s + type: LOGICAL_DNS + # Comment out the following line to test on v6 networks + dns_lookup_family: V4_ONLY + lb_policy: ROUND_ROBIN + load_assignment: + cluster_name: service_httpbin + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: httpbin.org + port_value: 80 + - name: rate_limit_cluster + type: LOGICAL_DNS + connect_timeout: 0.25s + lb_policy: ROUND_ROBIN + http2_protocol_options: {} + load_assignment: + cluster_name: rate_limit_cluster + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: sentinel-rls-service + port_value: 10245 +--- +apiVersion: apps/v1beta2 +kind: Deployment +metadata: + name: envoy-deployment-basic + labels: + app: envoy +spec: + replicas: 1 + selector: + matchLabels: + app: envoy + template: + metadata: + labels: + app: envoy + spec: + containers: + - name: envoy + image: envoyproxy/envoy:v1.12.0 + ports: + - containerPort: 10000 + command: ["envoy"] + args: ["-c", "/tmp/envoy/envoy.yaml"] + volumeMounts: + - name: envoy-config + mountPath: /tmp/envoy + volumes: + - name: envoy-config + configMap: + name: envoy-cm + items: + - key: envoy-yml + path: envoy.yaml +--- +apiVersion: v1 +kind: Service +metadata: + name: envoy-service + labels: + name: envoy-service +spec: + type: ClusterIP + ports: + - port: 10000 + targetPort: 10000 + protocol: TCP + selector: + app: envoy \ No newline at end of file diff --git a/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/sentinel-rls.yml b/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/sentinel-rls.yml new file mode 100644 index 00000000..1023932e --- /dev/null +++ b/sentinel-cluster/sentinel-cluster-server-envoy-rls/sample/k8s/sentinel-rls.yml @@ -0,0 +1,68 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: sentinel-rule-cm +data: + rule-yaml: |- + domain: foo + descriptors: + - resources: + - key: "destination_cluster" + value: "service_httpbin" + count: 1 +--- +apiVersion: apps/v1beta2 +kind: Deployment +metadata: + name: sentinel-rls-server + labels: + app: sentinel +spec: + replicas: 1 + selector: + matchLabels: + app: sentinel + template: + metadata: + labels: + app: sentinel + spec: + containers: + - name: sentinelserver + # You could replace the image with your own image here + image: "registry.cn-hangzhou.aliyuncs.com/sentinel-docker-repo/sentinel-envoy-rls-server:latest" + imagePullPolicy: Always + ports: + - containerPort: 10245 + - containerPort: 8719 + volumeMounts: + - name: sentinel-rule-config + mountPath: /tmp/sentinel + env: + - name: SENTINEL_RLS_RULE_FILE_PATH + value: "/tmp/sentinel/rule.yaml" + volumes: + - name: sentinel-rule-config + configMap: + name: sentinel-rule-cm + items: + - key: rule-yaml + path: rule.yaml +--- +apiVersion: v1 +kind: Service +metadata: + name: sentinel-rls-service + labels: + name: sentinel-rls-service +spec: + type: ClusterIP + ports: + - port: 8719 + targetPort: 8719 + name: sentinel-command + - port: 10245 + targetPort: 10245 + name: sentinel-grpc + selector: + app: sentinel \ No newline at end of file