What Is Traffic Mirroring (aka Replication) ?
Traffic replication is a process of making copies of incoming traffic. This could be all traffic coming into a particular host or only traffic coming to a particular port(s) on a host.
Fig 1: Traffic Replication
Why Replicate Traffic ?
For any web application it’s now a norm to have separate environments for Development, Testing/Staging/Pre-Production and Production. The utilization of the application is each environment is varied and the traffic pattern on each is different.
Fig 2: Application Environments
Fig 3: Environment Utilization
Even after all the performance and automated tests it’s still possible for new bugs to pop-up in production environment.
Common causes for the bugs :
- Unaccounted use cases during development.
- Invalid or new data in the incoming request.
- A certain traffic pattern or sequence of flows causing an issue.
Now, what if we had an environment (say pre-production) to which the replicated production traffic can be sent to. This environment would receive a wide variety of requests and also follow a traffic pattern similar to production. This makes it easier to validate any new code before pushing to production and also to troubleshoot any issues that pop up only in production environment and hard to reproduce in non-prod environments.
Fig 4: Replicate Production Traffic
How To Mirror(Replicate) Traffic In AWS ?
Approach #1 – Traffic Replication Using AWS VPC Mirroring
Fig 5: Architecture: VPC Traffic Mirroring
AWS takes the source traffic packet and wraps that as a VXLAN packet. The original packet is placed in the payload of the VXLAN packet. The target receiving the replicated packet has to unwrap the VXLAN packet and extract the original payload for processing.
Fig 6: VXLAN unwrap & Replay
Fig 7: GoLang Script – Part 1
Fig 8: GoLang Script – Part 2
Fig 9: GoLang Script – Part 3
- Non-intrusive way for replication traffic in an AWS environment. No need to install additional tools or packages on any host/EC2 instance.
- Live, low latency traffic replication
Approach #2 – Traffic Replication VIA A Third Party Tool
- Simple, easy and quick to setup and configure.
- Provides ability to filter traffic, rewrite request and replay as needed.
- Intrusive Approach. GoReplay needs to be installed and configured in EC2 instances that would be the source for replication.
- GoReplay will compete for CPU and Memory on the host machines, which indirectly may affect the application performance.
Traffic replication is an effective way of maintaining a near production like environment with minimal effort. The benefits of replication goes beyond just monitoring and troubleshooting issues. The replicated information can be used in performing data analysis, analyze traffic pattern and in detecting security vulnerabilities and attacks on the system.
In my opinion and based on personal experience I find the AWS VPC mirroring to be the best approach for replicating traffic for applications deployed on AWS.