Latency on a fiber channel network isn’t normally something you worry about. But something that you need to remember is that every meter of fiber optic cable that your data has to travel through takes time. Every fiber channel switch that you have to go through adds more latency.
When you are setting up something like synchronous replication between two storage arrays this latency starts to become more important as every millisecond that you spend waiting for the storage to respond is time that your application isn’t responding to your clients requests.
So, the basic math is that every meter of fiber optic that your data travels takes 5 nanoseconds. So if you have your server connected to your storage array via a one meter cable there will be 10 nanoseconds of delay. 5 nanoseconds for the data to get to the array, and 5 nanoseconds for the response to get back to the server from the array.
So using this math, for each 100 meters of fiber optic cable there is 1 microsecond of latency. For every kilometer of cable there is 10 microseconds. For every 100 kilometers of cable there is a 1 millisecond delay, and for every 1000 kilometers of cable there is 10 milliseconds of delay. So if you are replicating data from LA to New York that’s about 2778 miles, or 4470 kilometers which gives us a delay of about 44 milliseconds for each command which is being sent.
Now there is something else which needs to be taken into account when figuring out the storage latency time. And that is the fiber channel switches. If the ports on the fiber switch are on the same ASIC then there is no measurable latency through the switch, however if the two ports on the fiber switch are on different ASICs then there is an additional latency of 2 microseconds in each direction. While this isn’t much, if you keep in mind that between LA and New York there are probably hundreds of switches, those 2 microseconds are going to really start to add up.
Because of these numbers when using synchronous replication 30 miles is about as far as you want to replicate data. And farther than that and you’ll start to see network latency problems with your application. These problems will be amplified if you use something like SQL Server as with SQL Server and other databases, every nanosecond counts.
Hopefully this math will help you make more informed storage design decisions.
Denny
2 Responses
Actually your assumptions aren’t strictly true; bear in mind you won’t get dedicated lines that are 100km in length. Invariably fibre channel has to be routed over other equipment and those relaying devices add latency. Speed of Light is actually the least of your worries over long distance. What’s more important is the route and number of intermediate relay devices you have to negotiate. BTW, in that respect I’m not referring to FC switches, but things like DWDM equipment.
Chris Evans
Haven’t you done a calculation error?
First you say it takes 1 ms to go 100 km back and return – in total
Then you double it again going from LA to NY… shouldn’t the right answer be distance is 2778 so back and return is 27.78 ms ?
The 10 nanosecond for 1 meter is 5 nano one way and 5 nano return.