El Psy Congroo

Posted on

safepoint

Safepoints: Meaning, Side Effects and Overheads
Counted/Uncounted loops, Safepoints and OSR Compilation
JVM的Stop The World，安全点，黑暗的地底世界

When at a safepoint, the thread’s representation of it’s Java machine state is well described, and can be safely manipulated and observed by other threads in the JVM.

安全点是指线程执行过程中一些特定的位置，在这些位置上线程的JVM状态是确定的（JIT会在safepoint上生成OopMap帮助确定GC Roots），因此可以进行GC等操作

safepoint日志

  • 打印日志
    -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1
    -XX:+PrintGCApplicationStoppedTime #打印停顿时间

    Read more »
El Psy Congroo

Posted on

AAT

  • Rule for letter o
    • In a one-syllable word, o sounds like ä (unless the word ends in e)
      • hot, lost, Tom, Bob, dot com
    • In a stressed syllable, o also sounds like ä
      • possible, Holland, philosophy

  • Rule for letter a

    • In a one-syllable word, a sounds like æ (unless the word ends in e)
      • cat, Sam, drab
    • In a stressed syllable, a also sounds like æ
El Psy Congroo

Posted on

概述

Kafka本质上是一个append-only的日志系统，消息就是日志，按顺序追加到log文件中。为了能快速定位到消息，每个log文件还对应一个index文件，稀疏记录log文件中消息的offset，因此单个log文件也不能太大，log文件对应segment这个概念。为了提高并发性能，那就需要同时提供多组log文件以供写入，这就是topic中的partition概念，消息会hash到固定的partition中。为了保证高可用，每个partition都需要有备份，根据配置会复制1至N份到其他broker，保持同步的节点群共同组成ISR，当leader失效时从ISR中选出新leader。消息写入时，可以按对可靠性的要求，选择仅写入leader就返回，还是需要等待写入至少N个或全部replica。消息消费时，kafka使用pull的模式，服务端不负责记录消费者offset（但是提供了用于记录的基础设施，以前是zk，现在是特殊的topic），并且一个partition只允许一个消费者，这保证了消息是partition有序的。新版本kafka对zk的依赖很少，仅用于节点发现，集群协调工作都是由选举出的一台broker（controller）来负责的。kafka依赖操作系统中的基础设施来保证高性能，例如利用顺序读取的log文件来规避磁盘性能问题，pagecache做缓存，zero-copy提高发送性能，因此部署时也要考虑这些特性，避免使用raid（将顺序读写分散了），保留足够内存用于pagecache，选择合适的replica份数在空间和可靠性中取得均衡，规划带宽时不要忘了节点间同步的开销等。

Read more »
El Psy Congroo

Posted on

Spark

RDD (Resilient Distributed Datasets)

  • What is RDD

    RDDs are immutable, fault tolerant, parallel data structures that let users explicitly persist intermediate results in memory, control their partitioning to optimize data placement, and manipulate them using a rich set of operations.

    In a nutshell RDDs are a level of abstraction that enable efficient data reuse in a broad range of applications

    Read more »
El Psy Congroo

Posted on

Log4j 1.2

性能注意点

  • 使用Slf4j
  • 使用if (logger.isDebugEnabled()) { ... }封装String连接或其他高开销的操作
  • 避免多次打印Stack trace
  • 尽量避免使用类名%C, 文件名%F, 方法名%M, 行号%L，性能开销很大official warning
  • 通过缓存减少IO次数
    • immediateFlush
      • 是否每次log event都flush到磁盘，默认为true，设置为false能提高性能，但是在系统崩溃时可能会丢失日志
    • bufferedIO
      • 默认为false时使用java.io.Writer，设置为ture则改用BufferedWriter，bufferSize默认8K，可调整
  • 使用AsyncAppender
    • 使用addAppender封装普通Appender为异步
    • 通过独立的AsyncAppender-Dispatcher线程调用Appender
    • 通过bufferSize设置缓存的event数量，到上限时根据blocking参数（默认为true）决定阻塞还是丢弃后续event
      Read more »
El Psy Congroo

Posted on

持续更新中……

System Design

Basic Steps

  1. Clarify and agree on the scope of the system
    • User cases
      • Who is going to use it
      • How are they going to use it
    • Constraints
      • traffic
      • high availability
      • …
  2. High level architecture design (Abstract design)
    • Sketch the important components and connections between them, but don’t go into some details
      • service layer
      • data storage layer
      • key components, eg: caching, load balance, database cluster
  3. Component Design
    • interface
    • OOD for functionalities
      • Map scenario to module
  4. Understanding Bottlenecks
  5. Scaling
    • Vertical
    • Horizontal
    • Caching
      • Application caching
      • Database caching
      • In-memory caches
    • Asynchronized
      • Queue
      • Batch
    • Load balancing
      • client-side / hardware / software
    • Database partitioning
      • vertical / horizontal
    • Microservices
      Read more »