Why latency matters

We care a lot about latency at Codedose, and this is not only when building high performance computing processes but for any type of applications really.

Why is that?

We could observe a major shift in hard availability and its cost/performance ratio in the last 10 years, and while this also relates to processors the real impact is made by RAM memory and SSD storage devices.

Memory units became cheap, large and fast, and you can order real memory monsters with terabytes of RAM with a few clicks right now. Well, your credit card limit applies but joke asides – just see Dell PowerEdge R920 or HP ProLiant DL580. Yes, 6 TB of DDR4 RAM in one physical box at your service.

SSD storage devices are amazing at their speed, typically at least 20x faster than good old spinning disks. You can easily extend your desktop or laptop with Samsung 850 PRO disks or go with specialised SSD cards for your servers.

What does it mean to us, Java devs?

Just take a look at this brilliant summary of latency times required to transfer data between different physical units of your computer system: Latency Numbers Every Programmer Should Know:

Latency Comparison Numbers
 L1 cache reference 0.5 ns
 Branch mispredict 5 ns
 L2 cache reference 7 ns 14x L1 cache
 Mutex lock/unlock 25 ns
 Main memory reference 100 ns 20x L2 cache, 200x L1 cache
 Compress 1K bytes with Zippy 3,000 ns 3 us
 Send 1K bytes over 1 Gbps network 10,000 ns 10 us
 Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD
 Read 1 MB sequentially from memory 250,000 ns 250 us
 Round trip within same datacenter 500,000 ns 500 us
 Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us 1 ms ~1GB/sec SSD, 4X memory
 Disk seek 10,000,000 ns 10,000 us 10 ms 20x datacenter roundtrip
 Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms 80x memory, 20X SSD
 Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms

 1 ns = 10^-9 seconds
 1 us = 10^-6 seconds = 1,000 ns
 1 ms = 10^-3 seconds = 1,000 us = 1,000,000 ns

 By Jeff Dean: http://research.google.com/people/jeff/
 Originally by Peter Norvig: http://norvig.com/21-days.html#answers

Yes, main memory is 4x faster than SSD, and SSD is 20x faster than a spinning hard drive.

So what are the lessons for us?

  1. “Your data fits in RAM. Yes, it does.” – see RAM is the New SSD.
  2. Design your system to store all data in memory or cache aggressively otherwise. Observe and eliminate I/O hits.
  3. Process data in memory and leave the results there.
  4. Worried about persistency and transactional control? Try to restructure your processing to write data asynchronously into a persistent storage device and measure your throughput/latency twice – it may be that you can just reprocess whole batch once again in memory in case of any error.
  5. Really not enough memory to store your data? Or more CPU power needed to crunch your data? Look at distributed cache and grid-computing platforms like Oracle Coherence, Hazelcast or Spark.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s