zone aware routing: latency by zone + rps by zone

[greeshma1196]

This PR introduces two changes:

1. When I operate skipper-ingress controller across multiple availability zones, I want backend request latency labeled by the destination endpoint's zone, So that I can plot per-zone latency and RPS by zone

RPS by zone:

Latency by zone:

2. When I run skipper-ingress that consume routes from RouteSRV, I want each backend endpoint's availability zone to survive the eskip wire format, So that per-zone backend latency can be plotted when RouteSRV is used, the same as in the in-process kubernetes dataclient mode - ref: RouteSRV: keep zone information while serializing to eskip #4103

Result: In the test cluster, there were 2 instances of skipper-ingress one set to Zone-A, the other to Zone-B

route-dump of skipper-ingress in zone A

kube_rg__skipper_loadtest_target__loadtest_target__all__0_0: Host(/^(skipper-loadtest[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$/) && PathSubtree("/")
  -> disableAccessLog(2, 3, 404, 429)
  -> fifo(2000, 20, "1s")
  -> disableAccessLog()
  -> oauthTokeninfoValidate("{optOutAnnotations: [iam.zalando.org/public], optOutHosts: [\"^.*[.]ingress[.]cluster[.]local\"], unauthorizedResponse: \"Authentication required, see https://cloud.docs.zalando.net/howtos/authenticate-requests/#default-authentication\\n\"}")
  -> stateBagToTag("auth-user", "client.uid")
  -> <powerOfRandomNChoices, "http://[2a05:d024:90:1103:6d8f::4]:9090?zone=eu-central-1a", "http://[2a05:d024:90:1103:931f::b]:9090?zone=eu-central-1a", "http://[2a05:d024:90:1103:931f::c]:9090?zone=eu-central-1a">;

route-dump of skipper-ingress in zone B

kube_rg__skipper_loadtest_target__loadtest_target__all__0_0: Host(/^(skipper-loadtest[.]pg9-test[.]zalan[.]do[.]?(:[0-9]+)?)$/) && PathSubtree("/")
  -> disableAccessLog(2, 3, 404, 429)
  -> fifo(2000, 20, "1s")
  -> disableAccessLog()
  -> oauthTokeninfoValidate("{optOutAnnotations: [iam.zalando.org/public], optOutHosts: [\"^.*[.]ingress[.]cluster[.]local\"], unauthorizedResponse: \"Authentication required, see https://cloud.docs.zalando.net/howtos/authenticate-requests/#default-authentication\\n\"}")
  -> stateBagToTag("auth-user", "client.uid")
  -> <powerOfRandomNChoices, "http://[2a05:d024:90:1104:99f6::20]:9090?zone=eu-central-1b", "http://[2a05:d024:90:1104:99f6::24]:9090?zone=eu-central-1b", "http://[2a05:d024:90:1104:99f6::c]:9090?zone=eu-central-1b">;

[szuecs]

Rest is good to go from my side.

[szuecs]

👍

[szuecs]

If you want you can merge and create a new PR to refactor the const