云原生AI网关架构设计：从流量管控到大模型请求路由的全栈实现

1
2
3
4
5
6
7
8
9

// 限流规则示例
type RateLimitRule struct {
    UserID        string  // 用户ID
    Model         string  // 模型名称
    MaxTokensPerMinute int64 // 每分钟最大Token额度
    MaxConcurrent int     // 最大并发请求数
    MaxRequestTime int64   // 单个请求最大时长（秒）
    Priority      int     // 优先级
}

1
2
3
4
5

调度优先级排序：
1. 请求优先级（VIP用户 > 普通用户 > 测试用户）
2. 推理成本（优先调度到单位Token成本最低的实例）
3. 负载情况（优先调度到剩余算力最多的实例）
4. 地域亲和性（优先调度到离用户最近的实例）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-gateway
  namespace: ai-infra
  labels:
    app: ai-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-gateway
  template:
    metadata:
      labels:
        app: ai-gateway
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
    spec:
      containers:
      - name: ai-gateway
        image: registry.example.com/ai/ai-gateway:v1.2.0
        ports:
        - name: http
          containerPort: 8080
        - name: admin
          containerPort: 8081
        - name: metrics
          containerPort: 9090
        resources:
          requests:
            cpu: "2"
            memory: "4Gi"
          limits:
            cpu: "4"
            memory: "8Gi"
        livenessProbe:
          httpGet:
            path: /health/live
            port: admin
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/ready
            port: admin
          initialDelaySeconds: 10
          periodSeconds: 5
        env:
        - name: REDIS_ADDR
          value: "redis://ai-gateway-redis.ai-infra.svc.cluster.local:6379"
        - name: ETCD_ADDR
          value: "etcd://ai-infra-etcd.ai-infra.svc.cluster.local:2379"
        - name: OTL_ENDPOINT
          value: "http://otel-collector.observability.svc.cluster.local:4317"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

apiVersion: v1
kind: Service
metadata:
  name: ai-gateway
  namespace: ai-infra
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8080
  selector:
    app: ai-gateway

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-gateway
  namespace: ai-infra
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-gateway
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: active_requests
      target:
        type: AverageValue
        averageValue: 100

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: ai-gateway.example.com/v1alpha1
kind: RateLimitPolicy
metadata:
  name: vip-user-limit
spec:
  selector:
    userGroup: "vip"
  limits:
  - model: "gpt-4"
    maxTokensPerMinute: 100000
    maxConcurrent: 10
  - model: "gpt-3.5-turbo"
    maxTokensPerMinute: 1000000
    maxConcurrent: 50
  priority: 10