The system receives data from a third-party service using TCP sockets. The service requires a static IP address to send the data. Several app nodes are created to receive and process data. AWS ELB is used as the load balancer for app nodes. But currently AWS ELB only has host name, but no static IP address. AWS has elastic IP addresses which are static, but cannot be associated with ELB.
Solution #1 - HAProxy (Not working)
The first solution I had was to use a HAProxy server as the proxy to AWS ELB. Install HAProxy on one EC2 instance and assign an elastic IP address to it. HAProxy receives data and forward to ELB.
Issue with this solution is that HAProxy only resolves DNS names during start. So once HAProxy starts and IP address of ELB changes, there is no way to detect that and HAProxy keeps sending to old IP address.
Solution #2 - Synapse
Synapse is a service discovery system from Airbnb. It builds upon HAProxy. Synapse provides certain watchers which watch changes. Once changes are detected, Synapse generates a new HAProxy configuration and reloads HAProxy. Application talks to HAProxy instead of the actual proxied service.
Back to the problem, I used Synapse to replace ELB. Synapse has a watcher
ec2tag which can watch tags of EC2 instances. To add/remove instances from Synapse, just add/remove certain tags. For example, Synapse watches tag name/value
env=test of EC2 instances. Once a new instance with tag
env=test is launched, Synapse detects this change and update HAProxy config file to include the new instance. The new instance now is able to receive data. Load balancing is provided by HAProxy.
It's recommended to install Synapse directly from GitHub
master branch. Release
0.11.1 has some issues. For example, if you're using bundler, add following to your
gem 'synapse', :git => 'git://github.com/airbnb/synapse.git'
gem_specific_install "synapse" do repository "https://github.com/airbnb/synapse.git" revision "master" action :install end
Synapse configuration is done by a YAML file
synapse.conf.yaml. In this file, you define services and HAProxy configuration.
--- services: myservice: default_servers: - name: "elb" host: "<elb-host>" port: 7000 discovery: method: "ec2tag" tag_name: "env" tag_value: "test" aws_access_key_id: "<aws-key>" aws_secret_access_key: "<aws-secret>" aws_region: "<aws-region>" haproxy: port: 3200 server_port_override: "7000" server_options: "check inter 2000 rise 3 fall 2" frontend: - "mode tcp" backend: - "mode tcp" haproxy: bind_address: "0.0.0.0" reload_command: "service haproxy reload" config_file_path: "/etc/haproxy/haproxy.cfg" do_writes: true do_reloads: true global: - "log 127.0.0.1 local0" - "log 127.0.0.1 local1 notice" - "user haproxy" - "group haproxy" defaults: - "log global" - "balance roundrobin" - "timeout client 50s" - "timeout connect 5s" - "timeout server 50s"
In the Synapse config file above,
services section defines different services to watch. For
default_servers section contains the fallback servers when no servers are discovered, here I used the ELB server.
discovery section contains configuration about different discovery methods. For
ec2tag, you need to provide AWS access key & secret, region and tag name/value to watch.
haproxy section contains local HAProxy configuration for this service. In the example, HAProxy for
myservice listens on port
3200 and forwards traffic to app nodes no port
7000. The second
haproxy section contains global HAProxy configurations.
Copy the YAML file to some place, e.g.
/etc/synapse.conf.yaml, then start Synapse using
synapse -c /etc/synapse.conf.yaml.
Solution #3 Nginx (untested)
Nginx seems to have better support of DNS resolution, so it may work to use Nginx as the proxy.