Ultra Low Latency System

Latency Numbers for Programmers

Component Latency Comparison Numbers
L1 cache reference 0.5 ns
Branch mispredict 5 ns
L2 cache reference 7 ns 14x L1 cache
Mutex lock/unlock 25 ns
Main memory reference 100 ns 20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy 10,000 ns / 10 us
Send 1 KB bytes over 1 Gbps network 10,000 ns / 10 us
Read 4 KB randomly from SSD* 150,000 ns / 150 us ~1GB/sec SSD
Read 1 MB sequentially from memory 250,000 ns / 250 us
Round trip within same datacenter 500,000 ns / 500 us
Read 1 MB sequentially from SSD* 1,000,000 ns / 1,000 us / 1 ms ~1GB/sec SSD, 4X memory
HDD seek 10,000,000 ns / 10,000 us / 10 ms 20x datacenter roundtrip
Read 1 MB sequentially from 1 Gbps 10,000,000 ns / 10,000 us / 10 ms 40x memory, 10X SSD
Read 1 MB sequentially from HDD 30,000,000 ns / 30,000 us / 30 ms 120x memory, 30X SSD
Send packet CA->Netherlands->CA 150,000,000 ns / 150,000 us / 150 ms

Notes:

  • 1 ns = 10^-9 seconds
  • 1 us = 10^-6 seconds = 1,000 ns
  • 1 ms = 10^-3 seconds = 1,000 us = 1,000,000 ns

Is Computer Science a Science

  • Science seeks to explain phenomena through theory, hypothesis and experiment in an effort to ascertain natural laws
  • Engineer seeks to apply natural laws to the solution of practical problems
  • Computer Science is generally more engineering than science
  • Computer Science as applied to low latency programming is more science than engineering when you need to be able to hypothesis, measure and then explain: hypothesis, experiment, measure and draw conclusion

Types of Realtime: Determinism vs Absolute Latency

  1. Hard Real time: a pacemaker or weapons system
  2. First to the Bell: a cycling team
  3. Web Realtime: as long as the user doesn't notice a delay < 100 ms
  4. Soft Realtime: we care about all the latencies but can sacrifice the very furthest outliers for a better mean - most trading systems

Java

Language Comparison:

Language Latency
Java 10 uS
C/C++ 10 uS
FPGA 1uS
ASIC 400nS

Java is a reasonable choice for a low latency system in the > 10 uS range.

Response Time (Target Latency) Achievability Techniques
seconds (multi-second) easily achivable 1. small methods
2. minimize branching
3. use cohension
4. abstract cleanly
hundreds of ms (sub-second) general tactics as above plus:
1. optimized data structure
2. algorithm complexity minimized
3. use batching and caching
tens of ms (low) specific tactics as above plus:
1. memory access pattern optimized for CPU caches
2. lock-free algorithms
3. asynchronous processing
4. stateless / functional programming
5. RamFS/TmpFS
6. GC and object lifecycle tuning
under 1 ms (ultra low) very specific techniques as above plus:
1. thread affinity
2. NUMA
3. large pages
4. avoid false sharing
5. data-oriented design
6. disable c-states
7. ensure CPU cache friendly operations
8. no GC
9. zero copy

Challenges of Using Java

  • GC
  • Warmup
  • Unpredictable Compilation
  • No value types (C structs) - don't have control over the memory layout
  • Unnatural programming to get low level constructs (Unsafe)
  • Certain CPU instructions are not available

Realtime Contract Template

  • Activity A (very specific, e.g., logging)
  • When Run as part of system s
  • For a throughput of X event / second
  • For a duration of time T (including bursts)
  • On hardware H with operation system O
  • Should not exceed latency L @ percentile P
  • When correcting for coordinated omission

Hardest Areas in Realtime Systems

  • Wait-free (off-heap) data structures
  • Resilience / High Availability
  • Failover / Hot-Hot
  • Measurement

Area to Notice in realtime system of Microservice Architecture

  1. Fast Transport
    1. use shared memory: Chronicle Queue; Aeron
  2. Single Threaded
    1. Small JVM
    2. No synchronization
    3. No complicated object pooling strategy
    4. Scale out by partitioning your data and adding more microservices
  3. Hog the CPU
  4. Record the Input and Output
    1. Replay is especially important in low latency system
  5. Map your microservices onto your hardware

Finally, please check the video: How Low Can You Go

results matching ""

    No results matching ""