有很多小伙伴在小红书私信要大图，在这里分享出来。
如果觉得有帮助，请支持一下！
Boids algorithm is an artificial life algorithm proposed by Craig Reynolds in 1986, used to simulate the behavior of bird flocks, fish schools, etc. Boids algorithm simulates the interaction between individuals to achieve collective behavior of the group.
In the Boids algorithm, we have the following terminology:
In the Boids algorithm, we have the following rules:
Here is a demo of the Boids algorithm. You can adjust the parameters to see how the boids behave.
Actually, I think these three rules also exist in human society to some extent. So we can run some interesting experiments, such as:
Boids算法是Craig Reynolds在1986年提出的一种人工生命算法，用来模拟鸟群、鱼群等生物群体的行为。Boids算法通过模拟个体之间的互动，让整个群体表现出集体行为。
在Boids算法中，我们有以下术语：
在Boids算法中，我们有以下规则：
下面是Boids算法的演示。你可以调整参数，观察boid的行为变化。
我觉得这三条规则在一定程度上也存在于人类社会中，所以我们可以做一些有趣的思想实验，比如：
有很多小伙伴在小红书私信要大图，在这里分享出来。
如果觉得有帮助，请支持一下！
In this post, I summarize what I learned from the paper Time, Clocks, and the Ordering of Events in a Distributed System. Lamport published it in Communications of the ACM in 1978. The paper discusses time and clocks in distributed systems in depth. It proposes important concepts and algorithms such as “Happened Before”, logical clock, physical clock, and “State Machine”. It is a classic that every distributed systems reader must read.
Before we start, let’s consider the following problem:
Assume both are honest. Can we say that \(A\) happened before \(B\)? Clearly, we cannot tell which came first, because we don’t know whether their watches are “accurate”. What if the problem changes to the following?
We can see that clock readings and timestamps cannot describe the order of events accurately. Yet in a distributed system, the order of events often plays a “key role” in many algorithms. So how do we describe this order accurately, or design our clocks to avoid the problem in the example above?
The concept of time is fundamental to our way of thinking. It is derived from the more basic concept of the order in which events occur.
The definition of time is essential for understanding “the order of events” and “concurrency” in distributed systems. Lamport points out that time is derived from a more basic concept: the order in which events occur. For example, when we say something happened at 13:00, we mean it happened after we read 13:00 on the clock and before 13:01. So a clock is a way of discretizing continuous time by “numbering” it.
The distributed system in this post consists of several processes that are spatially separated. Events in the same process happen sequentially. Processes communicate by sending and receiving messages. A process can be an independent computer, an independent operating system process, or an independent hardware module inside one computer. From here on, I call a process a “node”. In particular, the communication delay between nodes is not negligible compared with the frequency of events inside a single node.
For events in a distributed system, we define the “happened before” relation, written as “\(\rightarrow\)”. It satisfies the following three conditions.
Two events are concurrent if and only if \(a \nrightarrow b, b \nrightarrow a\).
We also require \(\rightarrow\) to be irreflexive, that is, \(a \nrightarrow a\) . Clearly, it makes no sense to say that an event happened “before” itself.
To describe this relation intuitively, Lamport introduces the “space-time diagram” shown below. In the diagram, time runs upward along the vertical direction, and the horizontal direction shows different nodes in space. The black dots are events. The wavy arrows are messages.
Recall the “happened before” relation above. We can easily find event pairs in the diagram that satisfy it. For example, \(p_1 \rightarrow r_4\), which is derived from \(p_1 \rightarrow q_2 \rightarrow q_4 \rightarrow r_3 \rightarrow r_4\).
The diagram also has concurrent events, such as \(p_3\) and \(q_3\). In the diagram, we can see that \(p_3\) happens later than \(q_3\) in physical time. But the nodes in the system do not know which one came first.
Figure 1. space-time diagram
a clock is just a way of assigning a number to an event.
A clock is just a way of assigning a number to an event. More precisely, for each node \(P_i\), we define a clock \(C_i\) as a function. It assigns the number \(C_i \langle a \rangle\) to any event \(a\). For the whole system, the time of any event \(b\) is \(C \langle b \rangle\). If \(b\) happens on node \(P_j\), then \(C \langle b \rangle = C_j \langle b \rangle\). Here we treat the clock as a logical clock inside the system, not a physical clock. Its labels and counting method do not need to agree with physical time. To satisfy the “happened before” partial ordering above, our logical clock must satisfy the following Clock Condition.
Clock Condition. For any events \(a, b\) in the system: if \(a \rightarrow b\), then \(C \langle a \rangle < C \langle b \rangle\).
In particular, the converse of the Clock Condition, “if \(C \langle a \rangle < C \langle b \rangle\), then \(a \rightarrow b\)”, does not hold. It would require concurrent events to have the same logical time. For example, in Figure 1, both \(p_2\) and \(p_3\) are concurrent with \(q_3\). But by C1, \(C \langle p_2 \rangle < C \langle p_3 \rangle\). So we must have \(C \langle q_3 \rangle \neq C \langle p_2 \rangle\) or \(C \langle q_3 \rangle \neq C \langle p_3 \rangle\). This contradicts the concurrent relation.
For a logical clock, we can imagine “tick” events constantly happening inside a single node. For example, take two consecutive events \(a, b\) on the same node \(P_i\), with \(C_i \langle a \rangle = 4, C_i \langle b \rangle = 7\). Then tick events numbered \(5,6,7\) happen between them. So we can add “tick lines” to the space-time diagram, as the dashed lines in the figure below. By C1, there must be at least one tick line between two consecutive events on the same node. By C2, every message must cross at least one tick line.
Figure 2
To make this easier to understand, we can redraw the tick lines as equivalent horizontal lines, while keeping the partial ordering of events and messages, as in the figure below.
Figure 3
For the logical clock algorithm on a single node, we have the following Implementation Rules.
In practice, when we receive a message, we should run IR2 to update the time first. Then we run the actual event. This guarantees the Clock Condition.
With the logical clock, we can sort all events in the system into a total ordering. We first sort events by their time. For events with the same time, we use a priority \(\prec\) predefined on all nodes. The priority can be any rule, such as sorting by id.
More precisely, we define the total ordering \(\Rightarrow\). For event \(a\) on node \(P_i\) and event \(b\) on node \(P_j\), \(a \Rightarrow b\) if and only if (i) \(C_i \langle a \rangle < C_j \langle b \rangle\) or (ii) \(C_i \langle a \rangle = C_j \langle b \rangle\) and \(P_i \prec P_j\).
By the Clock Condition, anything that satisfies the partial ordering \(\rightarrow\) also satisfies the total ordering \(\Rightarrow\).
Figure 4. partial ordering and total ordering
Under the total ordering, events outside the system sometimes cause abnormal behavior.
Consider this case. Someone triggers event A on node A, then calls another person. After the call, that person triggers event B on node B. The system knows nothing about the outside event “phone call”. So it is possible to get \(B \Rightarrow A\).
We define the set of all events in the system as \(\varphi\). The set of system events together with external events is \(\underline{\varphi}\). \(\underline{\rightarrow}\) is the happened before relation on \(\underline{\varphi}\). In the example above, we have \(A \underline{\rightarrow} B\), but \(A \nrightarrow B\).
Clearly, no algorithm can guarantee the \(\underline{\rightarrow}\) relation from \(\varphi\) alone, without outside information. To ensure \(A \rightarrow B\), we have two options.
Strong Clock Condition. For any events \(a, b\) in \(\varphi\): if \(a \underline{\rightarrow} b\) then \(C \langle a \rangle < C \langle b \rangle\).
Clearly, compared with option 1, the Strong Clock Condition is the option we want. Next, I explain how to implement a physical clock that satisfies the Strong Clock Condition.
Let \(C_i(t)\) be the reading of clock \(C_i\) at physical time \(t\). For mathematical convenience, we assume \(C_i\) is continuously differentiable in \(t\). \(dC_i(t)/dt\) is the rate at which the clock runs at time \(t\).
For \(C_i\) to run at a rate close to real physical time, we need \(dC_i(t)/dt \approx 1\) for all \(t\). More precisely, we need the following condition.
Besides keeping each clock accurate, the clocks must also stay synchronized with each other. That is, \(C_i(t) \approx C_j(t)\) for all \(i,j,t\).
For PC2, because of accumulated error, two clocks that run completely independently will drift further and further apart. So we need an algorithm to synchronize the clocks on different nodes.
First, we assume our clocks satisfy the Clock Condition. Then we only need to consider the case \(a \nrightarrow b\) in \(\underline{\varphi}\). It is easy to see that \(a\) and \(b\) must happen on different nodes.
Let \(\mu\) be smaller than the minimum communication delay between nodes. That is, event \(a\) happens at physical time \(t\), and event \(b\) happens on another node. If \(a\underline{\rightarrow} b\), then \(b\) happens at \(t + \mu\) at the earliest. Usually, we can set \(\mu\) to the minimum distance between nodes divided by the speed of light.
To avoid the abnormal case above, we must make sure that \(C_i(t + \mu) - C_j(t) > 0\) for any \(i, j\) and \(t\) .
Combining with PC1, we have \(C_i(t + \mu) - C_i(t) > (1- \kappa)\mu\). See Appendix 8.2 for the derivation.
Combining with PC2, we need \(-\epsilon \geq -\mu(1 - \kappa)\), so we need \(\epsilon/(1 - \kappa) \leq \mu\). See Appendix 8.3 for the derivation.
Next, I describe the algorithm that makes the formulas above, PC1 and PC2 hold.
For a message \(m\) sent at physical time \(t\) and received at physical time \(t'\), we define the total delay of the message as \(v_m = t' - t\). The receiving node does not know the value of \(v_m\). But it can know the minimum delay of the message \(\mu_m\), where \(\mu_m \geq 0\) and \(\mu_m \leq v_m\). We call \(\xi_m = v_m - \mu_m\) the unpredictable delay.
For the physical clock algorithm on a single node, we have the following Implementation Rules.
Now we prove that the implementation rules above ensure PC2.
We view the whole system as a directed graph. The vertices are the nodes. A directed edge from \(P_i\) to \(P_j\) is a message link.
Let \(d\) be the diameter of the directed graph (longest shortest path). Let \(\tau\) be the minimum communication interval between two nodes. That is, between any time \(t\) and \(t + \tau\), \(P_i\) sends at least one message to \(P_j\). The theorem below tells us how long after system start-up the system reaches a time synchronization that satisfies PC2.
Theorem: Assume the system is a strongly connected graph with diameter \(d\) that follows IR1’ and IR2’. For any message \(m\), \(\mu_m \leq \mu\), where \(\mu\) is a specific constant, for all \(t \geq t_0\). (a) PC1 always holds. (b) There are constants \(\tau\) and \(\xi\) such that, on every edge of the system, a message with an unpredictable delay of at most \(\xi\) is sent every \(\tau\) seconds. Then PC2 is satisfied for all \(t\gtrapprox t_0 + \tau d\), with \(\epsilon \approx d(2\kappa\tau + \xi)\) and \(\mu + \xi \ll \tau\). The proof is in Appendix 8.4.
By PC1, \(\left \vert \frac{C_i(t+\mu) - C_i(t)}{\mu} < \kappa \right \vert\). So \((1 - \kappa)\mu < C_i(t+\mu) - C_i(t) < (1 + \kappa)\mu\) .
Continuing from 8.2, \(C_i(t) + \mu(1-\kappa) < C_i(t+\mu)\).
So for \(C_i(t + \mu) - C_j(t) > 0\), we need \(C_i(t) - C_j(t) > -\mu(1 - \kappa)\).
By PC2, \(C_i(t) - C_j(t) < -\epsilon\).
So we get \(\epsilon \leq \mu(1 - \kappa)\).
For any \(i\) and \(t\), we define \(C_i^t\) as a clock that is set to \(C_i\) at time \(t\), runs at the same rate as \(C_i\), and is never reset. That is,
\[C_i^t = C_i(t) + \int_{t}^{t'}[dC_i(t)/dt]dt \tag{1}\]For all \(t' \geq t\), because \(C_i\) can be reset, we note that
\[C_i(t') \geq C_i^t(t') \tag{2}\]Suppose \(P_1\) sends a message to \(P_2\) at time \(t_1\). \(P_2\) receives it at time \(t_2\). The unpredictable delay is \(\leq \xi\), and \(t_0 \leq t_1 \leq t_2\). Then for all \(t \geq t_2\), we have
\[\begin{aligned} & C_2^{t_2}(t) \geq C_2^{t_2}(t_2) + (1 - \kappa)(t - t_2) & \qquad [by\ (1)\ and\ PC1] \\ & \geq C_1(t_1) + \mu_m + (1 - \kappa)(t - t_2) & \qquad [by\ IR2'(b)] \\ & = C_1(t_1) + (1 - \kappa)(t - t_1) - [(t_2 - t_1) - \mu_m] + \kappa(t_2 - t_1) \\ & \geq C_1(t_1) + (1 - \kappa)(t - t_1) - \xi \end{aligned}\]So, using these assumptions, for all \(t \geq t_2\) we get
\[C_2^{t_2}(t) \geq C_1(t_1) + (1 - \kappa)(t - t_1) - \xi \tag{3}\]Now suppose that for \(i =1,...,n\), we have \(t_i \leq t_i' < t_{i+1}, t_0 \leq t_1\). At \(t_i'\), \(P_i\) sends a message to \(P_{i+1}\). The message is received at \(t_{i+1}\), and its unpredictable delay is less than \(\xi\). Applying equation (3) repeatedly, for \(t \geq t_{n+1}\) we get
\[C_{n+1}^{t_{n+1}}(t) \geq C_1(t_1') + (1 - \kappa)(t - t_1') - n\xi \tag{4}\]From PC1, IR1’, and IR2’, we can derive
\[C_1(t_1') \geq C_1(t_1) + (1 - \kappa)(t_1' - t_1)\]Combining this with (4) and (2), for \(t \geq t_{n+1}\) we get
\[C_{n+1}(t) \geq C_1(t_1) + (1 - \kappa)(t - t_1) - n\xi \tag{5}\]For any two nodes \(P\) and \(P'\), we can find a sequence of nodes \(P = P_0, P_1, ..., P_{n+1} = P', n \leq d\). By assumption (b), we can find times \(t_i, t_i'\) such that \(t_i' - t_i \leq \tau\) and \(t_{i+1} - t_i' \leq v\), where \(v=\mu + \xi\). So inequality (5) holds for any \(t \geq t_1 + d(\tau + v)\), with \(n \leq d\). For any \(i, j, t, t_1\) with \(t_1 \geq t_0\) and \(t \geq t_1 + d(\tau + v)\), we therefore have
\[C_i(t) \geq C_j(t_1) + (1 - \kappa)(t - t_1) - d\xi \tag{6}\]]]>本篇文章总结学习了Lamport于1978年发表在 Communications of the ACM 上的论文 Time, Clocks, and the Ordering of Events in a Distributed System，文章对分布式系统中的时间、时钟等概念做了深入的讨论，提出了“Happened Before”、“逻辑时钟”、“物理时钟”、“State Machine”等重要概念与算法，是分布式领域不能不读的经典论文。
在进入正题之前，首先让我们考虑如下问题：
假设两人都是诚实的，我们能说 \(A\) 发生在 \(B\) 之前吗？显然，由于我们并不知道两人的手表时间是否“准确”，所以不能准确地判断出孰先孰后。那么如果问题变成下面这样，结论又是如何呢？
由此可见，我们通常使用的时钟读数、时间戳等概念并不能准确地刻画出事件发生的先后顺序。而在分布式系统中，事件发生的先后常常扮演着各种算法的“关键角色”。那么我们如何准确地刻画这种顺序，或是设计我们的时钟，从而避免上面例子中的问题呢？
The concept of time is fundamental to our way of thinking. It is derived from the more basic concept of the order in which events occur.
时间的定义对于分布式领域关于“事件发生先后”、“并发”等概念的认识至关重要。Lamport在文中指出，时间是由更加基本的概念“事件发生的顺序”衍生出来的。例如我们说某件事在13:00发生，其实是在说这件事发生在我们读到时钟上的读数为13:00之后，13:01之前。由此可见，时钟其实是对连续的时间进行了离散化的“编号”。
本文中讨论的分布式系统，是由若干空间上分离的process组成。同一process上的事件顺序串行发生，process之间通过收发消息进行通信。这里的process可以是若干独立的计算机、独立的进程，或是一台计算机内独立的硬件模块。在后文中我们统称process为“节点”。特别地，各个节点之间的通信延迟与单个节点内部事件发生的频率相比，是不可忽略的。
对于一个分布式系统中的若干事件，我们定义“happened before”关系，用“\(\rightarrow\)”标识。其满足如下三个条件，
当且仅当 \(a \nrightarrow b, b \nrightarrow a\) 时，我们称两个事件为并发的(concurrent)。
此外，我们规定 \(\rightarrow\) 为非自反关系，即 \(a \nrightarrow a\) 。显然，说一件事发生在自己“之前”并无任何意义。
为了直观地描述这一关系，Lamport引入了如下图所示的“时空图”，图中垂直方向自下而上为时间先后顺序，水平方向为空间上的不同节点。图中的黑色圆点表示事件，波浪线箭头表示通信消息。
回顾上面的“happened before”关系，我们不难在图中找到若干满足条件的事件对，例如 \(p_1 \rightarrow r_4\)，其由 \(p_1 \rightarrow q_2 \rightarrow q_4 \rightarrow r_3 \rightarrow r_4\) 推导而来。
图中亦有若干并发的事件，例如 \(p_3\) 和 \(q_3\)，虽然在图中我们能看到 \(p_3\) 发生的物理时间(physical time)晚于 \(q_3\)，但对于系统中的节点来说，他们并不知道谁先谁后。
图1. space-time diagram
a clock is just a way of assigning a number to an event.
时钟仅仅是对事件的发生予以编号而已。 更加准确地讲，对于每一个节点 \(P_i\) 我们定义时钟 \(C_i\) 为一个函数，它为任意的事件 \(a\) 赋值编号为 \(C_i \langle a \rangle\)。对整个系统时钟来讲，任意事件 \(b\) 的发生时间标记为 \(C \langle b \rangle\)，如果其发生在节点 \(P_j\) 上，则 \(C \langle b \rangle = C_j \langle b \rangle\)。这里的时钟我们看做是系统内部的逻辑时钟，而非物理时钟，其标识与计数方法无需与物理时间一致。为了满足上文的”happened before”偏序关系，我们设计的逻辑时钟需要满足如下的Clock Condition。
Clock Condition. 对于系统中的任意事件 \(a, b\)：如果 \(a \rightarrow b\)，则 \(C \langle a \rangle < C \langle b \rangle\)。
特别地，Clock Condition的逆命题“如果 \(C \langle a \rangle < C \langle b \rangle\)，则 \(a \rightarrow b\)”并不成立。 因为它要求并发的事件必须具有相同的逻辑时间。例如图1中的 \(p_2,p_3\) 都与 \(q_3\) 为并发关系，但由 C1 有 \(C \langle p_2 \rangle < C \langle p_3 \rangle\)，则必然有 \(C \langle q_3 \rangle \neq C \langle p_2 \rangle\) 或 \(C \langle q_3 \rangle \neq C \langle p_3 \rangle\)，与并发关系矛盾。
对于逻辑时钟，我们可以想象单个节点内不断发生着“tick”事件，例如在同一节点 \(P_i\) 内连续发生的 \(a, b\) 两个事件，有 \(C_i \langle a \rangle = 4, C_i \langle b \rangle = 7\)，那么在这两个事件之间发生了编号为 \(5,6,7\) 的 tick 事件。于是我们可以在时空图中加入类似下图虚线所示的”tick line”。根据 C1 我们可以得到，在同一节点内的连续两个事件之间，至少要有一条 tick line。根据 C2 我们可以得到，每一条消息必须穿过至少一条 tick line。
图2
为了更方便理解，我们也可以在保证事件和消息的偏序关系下，将 tick line 绘制成如下图中等价的水平线的形式。
图3
对于单个节点上的逻辑时钟算法的实现，我们有如下的实现规则（Implementation Rule）：
在实践中，当我们收到某条消息后，应当先执行 IR2 修改时间，再执行具体事件，从而保证 Clock Condition。
利用逻辑时钟，我们可以对整个系统中的事件进行全序(total order)排序。我们首先根据事件发生的时间对其排序。对于发生时间相同的事件，我们引入对所有节点预先定义的优先级 \(\prec\)，这里的优先级可以是根据 id 排序等任意规则。
更加严谨地说，我们定义全序关系 \(\Rightarrow\)。对于发生在节点 \(P_i\) 的事件 \(a\) 和发生在节点 \(P_j\) 的事件 \(b\)，有 \(a \Rightarrow b\) 当且仅当 (i) \(C_i \langle a \rangle < C_j \langle b \rangle\) 或 (ii) \(C_i \langle a \rangle = C_j \langle b \rangle\) 且 \(P_i \prec P_j\)。
这里由 Clock Condition 我们可以看到，凡是满足偏序关系 \(\rightarrow\) 的，一定也满足全序关系 \(\Rightarrow\)。
图4. 偏序与全序关系
在全序关系下，由于系统之外的一些事件，使得我们有时会遇到一些反常行为。
考虑下面这种情况，某人在节点A上触发了事件A，随后打电话给另一个人。此人接到电话后在节点B上触发事件B。由于整个系统对于系统之外的事件“打电话”毫不知情，则有可能出现 \(B \Rightarrow A\) 的情况。
我们可以定义系统中的所有事件集合为 \(\varphi\)。系统中的事件与外部事件的合集为 \(\underline{\varphi}\)。\(\underline{\rightarrow}\) 为 \(\underline{\varphi}\) 上的 happened before 关系。在上面的例子中，我们有 \(A \underline{\rightarrow} B\)，但 \(A \nrightarrow B\)。
显然没有任何算法能够不利用外部信息，仅凭 \(\varphi\) 就能保证 \(\underline{\rightarrow}\) 关系。在此，为了能够确保\(A \rightarrow B\)，有如下两种方案，
Strong Clock Condition. 对于 \(\varphi\) 中的任意事件 \(a, b\)：如果\(a \underline{\rightarrow} b\) 则 \(C \langle a \rangle < C \langle b \rangle\)。
显然相较于方案1，Strong Clock Condition才是我们希望的方案。下面具体介绍如何实现满足Strong Clock Condition的物理时钟。
令 \(C_i(t)\) 表示时钟 \(C_i\) 在物理时间 \(t\) 读到的读数。为了数学上的方便，在此我们认为 \(C_i\) 对于 \(t\) 是连续可微的，\(dC_i(t)/dt\) 表示时钟在时间 \(t\) 运行的速率。
为了使 \(C_i\) 的运行速率与真实物理时钟相近，对于所有的 \(t\)，我们必须使得 \(dC_i(t)/dt \approx 1\)。更严谨地讲，我们需要满足如下条件，
除了保证单个时钟运行准确之外，各个时钟之间也需要保持同步，即对所有的 \(i,j,t\)，有 \(C_i(t) \approx C_j(t)\)。
对于 PC2，由于累计误差（accumulated error）的存在，两个完全独立运行的时钟必然会误差越来越大。因此我们需要某种算法对不同节点上的时钟进行对时。
首先我们假设我们的时钟满足Clock Condition，这样我们只需考虑在 \(\underline{\varphi}\) 中 \(a \nrightarrow b\) 的情况。不难发现，此时 \(a\) 与 \(b\) 必然发生在不同的节点上。
令 \(\mu\) 小于节点间的最小通信时延。即事件 \(a\) 发生于物理时间 \(t\)，事件 \(b\) 发生于另一节点，若 \(a\underline{\rightarrow} b\)，则 \(b\) 最早发生于 \(t + \mu\)。通常我们可以设定 \(\mu\) 为节点间的最小距离除以光速。
为了避免上文中的反常情况，我们必须保证对于任意的 \(i, j\) 和 \(t\) ，有 \(C_i(t + \mu) - C_j(t) > 0\)。
结合PC1，有 \(C_i(t + \mu) - C_i(t) > (1- \kappa)\mu\)，详细推导见附录8.2。
结合PC2，需保证 \(-\epsilon \geq -\mu(1 - \kappa)\)，则需有 \(\epsilon/(1 - \kappa) \leq \mu\)，详细推导见附录8.3。
下面介绍具体的算法实现，从而保证上面的公式与PC1，PC2成立。
对于一条发送于物理时间 \(t\) ，接收于物理时间 \(t'\) 的消息 \(m\)，我们定义消息的总延迟（total delay） \(v_m = t' - t\)。接收消息的节点当然不知道 \(v_m\) 的值，但是它可以知道这条消息的最小延迟（minimum delay） \(\mu_m\)， \(\mu_m \geq 0\) 且 \(\mu_m \leq v_m\)。我们称 \(\xi_m = v_m - \mu_m\) 为不可预测延迟（unpredictable delay）。
对于单个节点上的物理时钟算法的实现，我们有如下的实现规则（Implementation Rule）：
现在我们证明上述的实现规则可以确保满足PC2。
将整个系统视为一个有向图，图中的点为各个节点，\(P_i\) 到 \(P_j\) 的有向边视为其消息链路。
令 \(d\) 为有向图的直径(longest shortest path)。\(\tau\) 为两个节点之间的最低通信间隔，即任意时间 \(t\) 到 \(t + \tau\)之间，\(P_i\) 至少应该发送一条消息给 \(P_j\)。下面的定理给出系统启动后，至多多久系统会达成满足PC2的时间同步。
定理：假设系统为一个遵循IR1’和IR2’，且直径为 \(d\) 的强连通图。对于任意的消息 \(m\)，\(\mu_m \leq \mu\)，其中\(\mu\) 为某个特定常数，且对于所有 \(t \geq t_0\)。 (a) PC1 总是成立。 (b) 存在常数 \(\tau\) 和 \(\xi\)，在系统中每条边上，每\(\tau\)秒会转发一条不可预测延迟最大为 \(\xi\) 的消息。则PC2 满足于，对于所有的 \(t\gtrapprox t_0 + \tau d\)，\(\epsilon \approx d(2\kappa\tau + \xi)\)，\(\mu + \xi \ll \tau\)。其证明见于附录8.4。
由PC1有 \(\left \vert \frac{C_i(t+\mu) - C_i(t)}{\mu} < \kappa \right \vert\)，则有 \((1 - \kappa)\mu < C_i(t+\mu) - C_i(t) < (1 + \kappa)\mu\) 。
继8.2有，\(C_i(t) + \mu(1-\kappa) < C_i(t+\mu)\)。
故要使得 \(C_i(t + \mu) - C_j(t) > 0\)，则需有 \(C_i(t) - C_j(t) > -\mu(1 - \kappa)\)。
由PC2有，\(C_i(t) - C_j(t) < -\epsilon\)。
故而得，\(\epsilon \leq \mu(1 - \kappa)\)。
对于任意的 \(i\) 和 \(t\)，我们定义 \(C_i^t\) 为一个时钟，在时刻 \(t\) 设定为 \(C_i\) 并与 \(C_i\) 运行速率相同，且永不被修正(reset)。即，
\[C_i^t = C_i(t) + \int_{t}^{t'}[dC_i(t)/dt]dt \tag{1}\]对于所有的 \(t' \geq t\)，我们注意到由于 \(C_i\) 会被修正，有
\[C_i(t') \geq C_i^t(t') \tag{2}\]假设 \(P_1\) 在时刻 \(t_1\) 发送消息给 \(P_2\)，接收于时刻 \(t_2\)，不可预测延迟 \(\leq \xi\)，\(t_0 \leq t_1 \leq t_2\)。则对于所有的 \(t \geq t_2\)，我们有
\[\begin{aligned} & C_2^{t_2}(t) \geq C_2^{t_2}(t_2) + (1 - \kappa)(t - t_2) & \qquad [by\ (1)\ and\ PC1] \\ & \geq C_1(t_1) + \mu_m + (1 - \kappa)(t - t_2) & \qquad [by\ IR2'(b)] \\ & = C_1(t_1) + (1 - \kappa)(t - t_1) - [(t_2 - t_1) - \mu_m] + \kappa(t_2 - t_1) \\ & \geq C_1(t_1) + (1 - \kappa)(t - t_1) - \xi \end{aligned}\]因此，利用这些假设，我们可以得到，对于所有的\(t \geq t_2\)，有
\[C_2^{t_2}(t) \geq C_1(t_1) + (1 - \kappa)(t - t_1) - \xi \tag{3}\]现在假设对于 \(i =1,...,n\)，我们有\(t_i \leq t_i' < t_{i+1}, t_0 \leq t_1\)。在 \(t_i'\)时，\(P_i\) 发送一条消息给 \(P_{i+1}\)，该消息接收于 \(t_{i+1}\)，其不可预测延迟小于 \(\xi\)。反复应用等式(3)可以得到，对于 \(t \geq t_{n+1}\)，
\[C_{n+1}^{t_{n+1}}(t) \geq C_1(t_1') + (1 - \kappa)(t - t_1') - n\xi \tag{4}\]通过 PC1, IR1’, IR2’我们可以推导出，
\[C_1(t_1') \geq C_1(t_1) + (1 - \kappa)(t_1' - t_1)\]再结合(4)与(2)，我们可以得到对于 \(t \geq t_{n+1}\)，有
\[C_{n+1}(t) \geq C_1(t_1) + (1 - \kappa)(t - t_1) - n\xi \tag{5}\]对于任意的两个节点 \(P\) 和 \(P'\)，我们能找到一个节点序列 \(P = P_0, P_1, ..., P_{n+1} = P', n \leq d\)。通过假设 (b)，我们能找到时间 \(t_i, t_i'\)，有 \(t_i' - t_i \leq \tau\) 且 \(t_{i+1} - t_i' \leq v\)，其中 \(v=\mu + \xi\)。因此对于任何的 \(t \geq t_1 + d(\tau + v)\)，不等式 (5) 成立，且有 \(n \leq d\)。对于任意的\(i, j, t, t_1\)，其中 \(t_1 \geq t_0\) 且 \(t \geq t_1 + d(\tau + v)\)，我们因此有
\[C_i(t) \geq C_j(t_1) + (1 - \kappa)(t - t_1) - d\xi \tag{6}\]]]>In the previous post, libco Source Notes (2): Explicit Switching, we covered the explicit coroutine switching interfaces that libco provides, and discussed how to use a coroutine pool. This post covers the interfaces libco provides for automatic switching. I suggest reading it together with my annotated version.
In Li Fangyuan’s libco talk$^{[2]}$, he explains that before libco, most network communication in WeChat used synchronous IO interfaces. To quickly adapt the existing business code, libco hooks system calls and provides coroutine-based primitives such as poll, read and write. Because of how coroutines work, a system call that used to block now behaves as if it were non-blocking.
libco needs to manage events that require timeouts, such as network IO and condition variables, in one place. For this, it implements a timeout manager based on a timing wheel. Before I cover how libco hooks system calls, let’s lay some groundwork on how this timeout manager is implemented.
As Figure 1 shows, the timing wheel is the dark red wheel-shaped array in the figure. we call each cell of the array a slot. A single slot stores the list of events registered within a certain period of time (the yellow linked lists in the figure). In libco, one slot has a precision of 1 millisecond. The whole wheel has 60000 slots, so it covers 60 seconds. The timing wheel has two main interface functions in libco, shown below.
AddTimeout computes the difference between the current time allNow and the wheel start time ullStart, and inserts the item into the matching slot. Note that when the timeout is longer than the wheel length of 60 seconds, libco inserts the event into the “last” slot.
TakeAllTimeout computes the difference between the current time allNow and the wheel start time ullStart to find the matching slot. It then walks all slots from the slot at the start index ullStartIdx up to that slot, and moves the timed-out items into the result list apResult.
With this, we can see that libco uses the timing wheel to manage timeout events efficiently.
/* Insert a new item into the timing wheel
* @param
* apTimeout :timing wheel structure
* apItem :new timeout item
* allNow :current time (timestamp in ms)
* @return :0 on success, else the failure line number
*/
int AddTimeout( stTimeout_t *apTimeout,stTimeoutItem_t *apItem ,unsigned long long allNow );
/* Take all timeout items out of the timing wheel
* @param
* apTimeout:timing wheel structure
* allNow :current time (timestamp in ms)
* apResult :result list of timeout events
*/
inline void TakeAllTimeout( stTimeout_t *apTimeout,unsigned long long allNow,stTimeoutItemLink_t *apResult );
Figure 1. Timeout management
libco manages IO events with epoll. co_eventloop triggers the IO events and switches to the matching coroutine to run it. Recall the thread-private global variable stCoRoutineEnv_t mentioned earlier, which represents the coroutine runtime environment. It holds the epoll structure handle pEpoll:
struct stCoRoutineEnv_t
{
stCoRoutine_t *pCallStack[ 128 ]; //call stack of all coroutines
int iCallStackSize; //index of the top of pCallStack
stCoEpoll_t *pEpoll; //epoll wrapper
//for copy stack log lastco and nextco
stCoRoutine_t* pending_co;
stCoRoutine_t* occupy_co; //current coroutine
};
stCoEpoll_t is defined as follows:
struct stCoEpoll_t
{
int iEpollFd; //EpollFd
static const int _EPOLL_SIZE = 1024 * 10; //max number of events returned by one epoll_wait call
struct stTimeout_t *pTimeout; //timing wheel, timeout management
struct stTimeoutItemLink_t *pstTimeoutList; //list of timed-out items
struct stTimeoutItemLink_t *pstActiveList; //list of ready items
co_epoll_res *result; //epoll_wait result
};
The main event loop code is as follows,
$L11$ blocks on epoll_wait with a timeout of 1 millisecond. This epoll_wait is not hooked. It is the native system call.
$L13-L29$ takes out all events in result->events, runs the prepare function pfnPrepare, and adds them to the active list.
$L32-L42$ takes out all timeout events and adds them to the active list.
$L59$ calls the process function pfnProcess for every event in the active list.
$L66$ checks whether the event loop needs to exit:
/* Event loop
* @param
* ctx:epoll handle
* pfn:function that checks whether to exit the event loop
* arg:argument of pfn
*/
void co_eventloop( stCoEpoll_t *ctx,pfn_co_eventloop_t pfn,void *arg )
{
if( !ctx->result )
{
ctx->result = co_epoll_res_alloc( stCoEpoll_t::_EPOLL_SIZE );
}
co_epoll_res *result = ctx->result;
for(;;)
{
int ret = co_epoll_wait( ctx->iEpollFd,result,stCoEpoll_t::_EPOLL_SIZE, 1 );
stTimeoutItemLink_t *active = (ctx->pstActiveList);
stTimeoutItemLink_t *timeout = (ctx->pstTimeoutList);
memset( timeout,0,sizeof(stTimeoutItemLink_t) ); //clear the timeout queue
for(int i=0;i<ret;i++) //walk the fds that have events
{
stTimeoutItem_t *item = (stTimeoutItem_t*)result->events[i].data.ptr; //get the stTimeoutItem_t that the event data points to
if( item->pfnPrepare ) //if there is a prepare function, run it; it adds the item to the ready list
{
item->pfnPrepare( item,result->events[i],active );
}
else //add to the ready list manually
{
AddTail( active,item );
}
}
unsigned long long now = GetTickMS();
TakeAllTimeout( ctx->pTimeout,now,timeout ); //insert the timed-out items into the timeout list
stTimeoutItem_t *lp = timeout->head;
while( lp )
{
//printf("raise timeout %p\n",lp);
lp->bTimeout = true; //mark as timed out
lp = lp->pNext;
}
Join<stTimeoutItem_t,stTimeoutItemLink_t>( active,timeout ); //merge the timeout list into the ready list
lp = active->head;
while( lp )
{
PopHead<stTimeoutItem_t,stTimeoutItemLink_t>( active );
if (lp->bTimeout && now < lp->ullExpireTime) //marked as timed out but the expire time is not reached yet, add it back to the timing wheel
{
int ret = AddTimeout(ctx->pTimeout, lp, now);
if (!ret)
{
lp->bTimeout = false;
lp = active->head;
continue;
}
}
if( lp->pfnProcess ) //call the process function of the stTimeoutItem_t item
{
lp->pfnProcess( lp );
}
lp = active->head;
}
if( pfn ) //lets the user break out of the event loop
{
if( -1 == pfn( arg ) )
{
break;
}
}
}
}
pollstruct pollfd {
int fd; /* file descriptor */
short events; /* requested events */
short revents; /* returned events */
};
int poll(struct pollfd *fds, nfds_t nfds, int timeout);
The original poll function takes the fds and the events of interest events as an array of pollfd. It returns the events that happened in revents. poll also supports a timeout in milliseconds. When timeout is set to a non-zero value (a negative value means forever), the thread blocks in poll until a matching event happens or the timeout expires.
The hooked poll in libco can give up the context when an IO coroutine blocks, and switch to the main coroutine. Most of its code merges and restores the events of identical fds in the pollfd array passed in. The core part of the code is co_poll_inner, shown below:
/* poll core
* @param
* ctx:epoll handle
* fds:fd array
* nfds:length of the fd array
* timeout:timeout in ms
* pollfunc:default poll
*/
int co_poll_inner( stCoEpoll_t *ctx,struct pollfd fds[], nfds_t nfds, int timeout, poll_pfn_t pollfunc)
{
if (timeout == 0) //poll: Specifying a timeout of zero causes poll() to return immediately, even if no file descriptors are ready.
{
return pollfunc(fds, nfds, timeout); //call the native system poll (the upper-level poll has already checked this, so it is not needed here)
}
if (timeout < 0) //poll: Specifying a negative value in timeout means an infinite timeout.
{
timeout = INT_MAX;
}
int epfd = ctx->iEpollFd;
stCoRoutine_t* self = co_self();
//1.struct change
stPoll_t& arg = *((stPoll_t*)malloc(sizeof(stPoll_t))); //allocate a stPoll_t
memset( &arg,0,sizeof(arg) );
arg.iEpollFd = epfd; //link stPoll_t with stCoEpoll_t here
arg.fds = (pollfd*)calloc(nfds, sizeof(pollfd)); //allocate nfds pollfd
arg.nfds = nfds;
stPollItem_t arr[2];
if( nfds < sizeof(arr) / sizeof(arr[0]) && !self->cIsShareStack) //when nfds is less than 2 and the shared stack is not used
{
arg.pPollItems = arr;
}
else
{
arg.pPollItems = (stPollItem_t*)malloc( nfds * sizeof( stPollItem_t ) );
}
memset( arg.pPollItems,0,nfds * sizeof(stPollItem_t) );
arg.pfnProcess = OnPollProcessEvent; //process function, calls co_resume(arg.pArg) to wake up the coroutine that arg.pArg points to
arg.pArg = GetCurrCo( co_get_curr_thread_env() ); //argument of the process function, i.e. the current coroutine
//2. add epoll
for(nfds_t i=0;i<nfds;i++)
{
arg.pPollItems[i].pSelf = arg.fds + i; //link stPollItem_t with pollfd
arg.pPollItems[i].pPoll = &arg; //point to the stPoll_t it belongs to
arg.pPollItems[i].pfnPrepare = OnPollPreparePfn; //set the prepare function
struct epoll_event &ev = arg.pPollItems[i].stEvent;
if( fds[i].fd > -1 ) //fd is valid
{
ev.data.ptr = arg.pPollItems + i; //make stPollItem_t.stEvent.data.ptr point to stPollItem_t
ev.events = PollEvent2Epoll( fds[i].events ); //set stPollItem_t.stEvent.data.events
int ret = co_epoll_ctl( epfd,EPOLL_CTL_ADD, fds[i].fd, &ev ); //add stPollItem_t.stEvent to stCoEpoll_t.iEpollFd
if (ret < 0 && errno == EPERM && nfds == 1 && pollfunc != NULL) //when nfds is 1 and adding to epoll fails, free the temporary stPoll_t
{
if( arg.pPollItems != arr )
{
free( arg.pPollItems );
arg.pPollItems = NULL;
}
free(arg.fds);
free(&arg);
return pollfunc(fds, nfds, timeout); //run the native poll
}
}
//if fail,the timeout would work
}
//3.add timeout
unsigned long long now = GetTickMS();
arg.ullExpireTime = now + timeout;
int ret = AddTimeout( ctx->pTimeout,&arg,now ); //add stPoll_t to the timing wheel of stCoEpoll_t
int iRaiseCnt = 0;
if( ret != 0 )
{
co_log_err("CO_ERR: AddTimeout ret %d now %lld timeout %d arg.ullExpireTime %lld",
ret,now,timeout,arg.ullExpireTime);
errno = EINVAL;
iRaiseCnt = -1;
}
else
{
co_yield_env( co_get_curr_thread_env() ); //give up the CPU and wait for an event in epoll or a timeout
iRaiseCnt = arg.iRaiseCnt; //we are back; before coming back, OnPollPreparePfn has already set iRaiseCnt and revents on stPoll_t, and removed it from the timing wheel
}
{
//clear epoll status and memory
RemoveFromLink<stTimeoutItem_t,stTimeoutItemLink_t>( &arg ); //remove from the timing wheel
for(nfds_t i = 0;i < nfds;i++)
{
int fd = fds[i].fd;
if( fd > -1 )
{
co_epoll_ctl( epfd,EPOLL_CTL_DEL,fd,&arg.pPollItems[i].stEvent ); //remove from epoll
}
fds[i].revents = arg.fds[i].revents; //return the events that have fired
}
if( arg.pPollItems != arr ) //free stPoll_t
{
free( arg.pPollItems );
arg.pPollItems = NULL;
}
free(arg.fds);
free(&arg);
}
return iRaiseCnt;
}
As the figure below shows, the first half of the code, $L1-L90$, wraps the fds in stPoll_t and adds it to the timing wheel timeout manager.
$L91$ gives up the CPU and switches out of the current coroutine.
After $L92$, once the event loop fires, it calls the OnPollProcessEvent set at L41 to switch to the matching coroutine, and then does the cleanup:
void OnPollProcessEvent( stTimeoutItem_t * ap )
{
stCoRoutine_t *co = (stCoRoutine_t*)ap->pArg;
co_resume( co );
}
Figure 2. Structures related to poll
This post covered the parts of the libco source code that handle automatic switching. Thank you for reading. If you have any questions or thoughts, or find any mistake in this post, please let me know.
在之前的文章libco源码笔记(2)显式切换中，我们介绍了libco提供的显式协程切换接口，并讨论了协程池的使用。本文讨论libco提供的自动切换相关函数接口。建议配合我自己的注释版本阅读本文。
李方源的libco分享$^{[2]}$中讲到，使用libco之前，微信大多数的网络通信使用同步IO接口。为了快速改造现有业务代码，libco以hook系统调用的形式，提供了协程基础上的poll，read，write等原语。利用协程的特性，原来阻塞的系统调用可以达到非阻塞的效果。
libco为了统一管理网络IO、条件变量等需要超时管理的事件，实现了基于时间轮(timing wheel)的超时管理器。在介绍其对系统调用的hook前，让我们先铺垫一些关于这个超时管理器的实现。
如下图1所示，时间轮为图中深红色的轮状数组，数组的每一个单元我们称为一个槽(slot)。单个slot里存储一定时间内注册的事件列表（图中黄色链表）。在libco中，单个slot的精度为1毫秒，整个时间轮由60000个slot组成，对应的整个时间轮覆盖60秒的时间。libco中关于时间轮的接口函数主要是下面两个。
AddTimeout通过计算当前时间allNow与时间轮起始时间ullStart的差，插入对应slot。特别注意的是当超时事件大于轮长60秒时，libco将这种事件插入到“最后一个”slot。
TakeAllTimeout通过计算当前时间allNow与时间轮起始时间ullStart的差，得出对应slot，遍历从起始索引ullStartIdx所指slot到该slot之间的所有slot中的超时项并移动到结果链表apResult中。
至此，我们看到通过时间轮，libco得以高效地完成对超时事件的管理。
/* 在时间轮中插入新项
* @param
* apTimeout :时间轮结构
* apItem :新的超时项
* allNow :当前时间(timestamp in ms)
* @return :0成功, else失败行数
*/
int AddTimeout( stTimeout_t *apTimeout,stTimeoutItem_t *apItem ,unsigned long long allNow );
/* 在时间轮中取出所有超时项
* @param
* apTimeout:时间轮结构
* allNow :当前时间(timestamp in ms)
* apResult :超时事件结果链表
*/
inline void TakeAllTimeout( stTimeout_t *apTimeout,unsigned long long allNow,stTimeoutItemLink_t *apResult );
图1. 超时管理
libco通过epoll管理IO事件，通过co_eventloop触发IO事件，并切换至对应协程执行。我们回顾之前提到过的、表示协程运行环境的线程私有全局变量stCoRoutineEnv_t，其中持有epoll的结构体句柄pEpoll。
struct stCoRoutineEnv_t
{
stCoRoutine_t *pCallStack[ 128 ]; //所有协程的调用栈
int iCallStackSize; //pCallStack栈顶索引
stCoEpoll_t *pEpoll; //epoll封装
//for copy stack log lastco and nextco
stCoRoutine_t* pending_co;
stCoRoutine_t* occupy_co; //当前协程
};
stCoEpoll_t的定义如下，
struct stCoEpoll_t
{
int iEpollFd; //EpollFd
static const int _EPOLL_SIZE = 1024 * 10; //epoll_wait单次最大返回事件数量
struct stTimeout_t *pTimeout; //时间轮, 超时管理
struct stTimeoutItemLink_t *pstTimeoutList; //已超时项链表
struct stTimeoutItemLink_t *pstActiveList; //已就绪项链表
co_epoll_res *result; //epoll_wait结果
};
主事件循环代码如下，
$L11$阻塞在epoll_wait上，并设置超时时间为1毫秒。这里的epoll_wait并没有经过hook，是系统原生的。
$L13-L29$取出所有的result->events的事件，执行预处理函数pfnPrepare，并加入active链表。
$L32-L42$取出所有超时事件并加入active链表。
$L59$对所有的active链表中事件调用处理函数pfnProcess。
$L66$检查是否需要退出事件循环。
/* 事件循环
* @param
* ctx:epoll句柄
* pfn:退出事件循环检查函数
* arg:pfn参数
*/
void co_eventloop( stCoEpoll_t *ctx,pfn_co_eventloop_t pfn,void *arg )
{
if( !ctx->result )
{
ctx->result = co_epoll_res_alloc( stCoEpoll_t::_EPOLL_SIZE );
}
co_epoll_res *result = ctx->result;
for(;;)
{
int ret = co_epoll_wait( ctx->iEpollFd,result,stCoEpoll_t::_EPOLL_SIZE, 1 );
stTimeoutItemLink_t *active = (ctx->pstActiveList);
stTimeoutItemLink_t *timeout = (ctx->pstTimeoutList);
memset( timeout,0,sizeof(stTimeoutItemLink_t) ); //清空超时队列
for(int i=0;i<ret;i++) //遍历有事件的fd
{
stTimeoutItem_t *item = (stTimeoutItem_t*)result->events[i].data.ptr; //获取event里数据指向的stTimeoutItem_t
if( item->pfnPrepare ) //如果有预处理函数，执行，由其加入就绪列表
{
item->pfnPrepare( item,result->events[i],active );
}
else //手动加入就绪列表
{
AddTail( active,item );
}
}
unsigned long long now = GetTickMS();
TakeAllTimeout( ctx->pTimeout,now,timeout ); //将超时项插入超时列表
stTimeoutItem_t *lp = timeout->head;
while( lp )
{
//printf("raise timeout %p\n",lp);
lp->bTimeout = true; //设置为超时
lp = lp->pNext;
}
Join<stTimeoutItem_t,stTimeoutItemLink_t>( active,timeout ); //将超时列表合并入就绪列表
lp = active->head;
while( lp )
{
PopHead<stTimeoutItem_t,stTimeoutItemLink_t>( active );
if (lp->bTimeout && now < lp->ullExpireTime) //还未达到超时时间但已经标记为超时的，加回时间轮
{
int ret = AddTimeout(ctx->pTimeout, lp, now);
if (!ret)
{
lp->bTimeout = false;
lp = active->head;
continue;
}
}
if( lp->pfnProcess ) //调用stTimeoutItem_t项的执行函数
{
lp->pfnProcess( lp );
}
lp = active->head;
}
if( pfn ) //用于用户控制跳出事件循环
{
if( -1 == pfn( arg ) )
{
break;
}
}
}
}
pollstruct pollfd {
int fd; /* file descriptor */
short events; /* requested events */
short revents; /* returned events */
};
int poll(struct pollfd *fds, nfds_t nfds, int timeout);
原始的poll函数以pollfd数组的形式传入fd以及关注的事件events，并将发生的事件返回到revents中。此外poll支持毫秒级别的超时设置，当timeout设置为非0值时(负数为永久)，线程会阻塞于poll直到有对应事件发生或超时。
libco hook后的poll可以在IO协程阻塞时让出上下文，切换至主协程。其中，大部分代码是处理传入pollfd数组中相同fd的事件合并和还原。代码中的核心部分为co_poll_inner，其代码如下，
/* poll内核
* @param
* ctx:epoll句柄
* fds:fd数组
* nfds:fd数组长度
* timeout:超时时间ms
* pollfunc:默认poll
*/
int co_poll_inner( stCoEpoll_t *ctx,struct pollfd fds[], nfds_t nfds, int timeout, poll_pfn_t pollfunc)
{
if (timeout == 0) //poll: Specifying a timeout of zero causes poll() to return immediately, even if no file descriptors are ready.
{
return pollfunc(fds, nfds, timeout); //调用系统原生poll(其实上层poll已经做过检查了，此处无需再做)
}
if (timeout < 0) //poll: Specifying a negative value in timeout means an infinite timeout.
{
timeout = INT_MAX;
}
int epfd = ctx->iEpollFd;
stCoRoutine_t* self = co_self();
//1.struct change
stPoll_t& arg = *((stPoll_t*)malloc(sizeof(stPoll_t))); //分配一个stPoll_t
memset( &arg,0,sizeof(arg) );
arg.iEpollFd = epfd; //此处stPoll_t与stCoEpoll_t进行关联
arg.fds = (pollfd*)calloc(nfds, sizeof(pollfd)); //分配nfds个pollfd
arg.nfds = nfds;
stPollItem_t arr[2];
if( nfds < sizeof(arr) / sizeof(arr[0]) && !self->cIsShareStack) //nfds少于2且未使用共享栈的情况下
{
arg.pPollItems = arr;
}
else
{
arg.pPollItems = (stPollItem_t*)malloc( nfds * sizeof( stPollItem_t ) );
}
memset( arg.pPollItems,0,nfds * sizeof(stPollItem_t) );
arg.pfnProcess = OnPollProcessEvent; //处理函数, 调用co_resume(arg.pArg), 唤醒参数arg.pArg所指协程
arg.pArg = GetCurrCo( co_get_curr_thread_env() ); //处理函数参数, 即当前协程
//2. add epoll
for(nfds_t i=0;i<nfds;i++)
{
arg.pPollItems[i].pSelf = arg.fds + i; //关联stPollItem_t与pollfd
arg.pPollItems[i].pPoll = &arg; //指向所属stPoll_t
arg.pPollItems[i].pfnPrepare = OnPollPreparePfn; //设置预处理
struct epoll_event &ev = arg.pPollItems[i].stEvent;
if( fds[i].fd > -1 ) //fd有效
{
ev.data.ptr = arg.pPollItems + i; //设置stPollItem_t.stEvent.data.ptr指向stPollItem_t
ev.events = PollEvent2Epoll( fds[i].events ); //设置stPollItem_t.stEvent.data.events
int ret = co_epoll_ctl( epfd,EPOLL_CTL_ADD, fds[i].fd, &ev ); //将stPollItem_t.stEvent加入stCoEpoll_t.iEpollFd中
if (ret < 0 && errno == EPERM && nfds == 1 && pollfunc != NULL) //nfds只有一个时，插入epoll失败, 释放掉临时的stPoll_t
{
if( arg.pPollItems != arr )
{
free( arg.pPollItems );
arg.pPollItems = NULL;
}
free(arg.fds);
free(&arg);
return pollfunc(fds, nfds, timeout); //执行原生poll
}
}
//if fail,the timeout would work
}
//3.add timeout
unsigned long long now = GetTickMS();
arg.ullExpireTime = now + timeout;
int ret = AddTimeout( ctx->pTimeout,&arg,now ); //将stPoll_t加入stCoEpoll_t的时间轮
int iRaiseCnt = 0;
if( ret != 0 )
{
co_log_err("CO_ERR: AddTimeout ret %d now %lld timeout %d arg.ullExpireTime %lld",
ret,now,timeout,arg.ullExpireTime);
errno = EINVAL;
iRaiseCnt = -1;
}
else
{
co_yield_env( co_get_curr_thread_env() ); //让出CPU, 等待epoll中的事件发生或超时
iRaiseCnt = arg.iRaiseCnt; //再次回来, 回来前会执行OnPollPreparePfn, 已经将stPoll_t设置好iRaiseCnt, revents, 并从时间轮中删除
}
{
//clear epoll status and memory
RemoveFromLink<stTimeoutItem_t,stTimeoutItemLink_t>( &arg ); //从时间轮中删除
for(nfds_t i = 0;i < nfds;i++)
{
int fd = fds[i].fd;
if( fd > -1 )
{
co_epoll_ctl( epfd,EPOLL_CTL_DEL,fd,&arg.pPollItems[i].stEvent ); //从epoll中删除
}
fds[i].revents = arg.fds[i].revents; //返回已经触发的事件
}
if( arg.pPollItems != arr ) //释放stPoll_t
{
free( arg.pPollItems );
arg.pPollItems = NULL;
}
free(arg.fds);
free(&arg);
}
return iRaiseCnt;
}
如下图所示，代码中前半部分$L1-L90$将fd封装为stPoll_t，并将其加入到时间轮超时管理器中。
$L91$让出CPU，切出当前协程。
$L92$之后是在事件循环触发后，调用L41中设置的OnPollProcessEvent切换至对应协程，然后进行对应的清理工作。
void OnPollProcessEvent( stTimeoutItem_t * ap )
{
stCoRoutine_t *co = (stCoRoutine_t*)ap->pArg;
co_resume( co );
}
图2. poll相关结构
至此，我们介绍了libco源码中自动切换的部分代码。感谢你的阅读。如果你有任何疑问和感想，或发现本文有任何错误，请一定让我知道。
In the previous post, libco Source Notes (1): Coroutines and Context Switching, we covered the basic concept of coroutines and the core context-switching code in libco. This post uses an example to introduce several important function interfaces that libco provides. I suggest reading it together with my annotated version.
First, we look at the three core structures in libco. Figure 1 below shows how they relate to each other.
coctx_tIt saves the context information needed when switching coroutines. For details, see libco Source Notes (1): Coroutines and Context Switching. I do not repeat them here.
stCoRoutine_tThis is the main coroutine structure. It holds all the information about a single coroutine, such as its start/stop state, its function, its context, and its shared stack.
stCoRoutineEnv_tstatic __thread stCoRoutineEnv_t* gCoEnvPerThread = NULL; //Coroutine runtime environment. __thread: thread-private
This is a thread-private global static variable. It holds the global coroutine environment, such as the coroutine call stack and the epoll handle. pCallStack is the coroutine call stack of the current thread. Because libco uses asymmetric coroutines
Figure 1. libco core structures
/* Coroutine creation interface
* @param
* co :double pointer to the main coroutine structure
* attr :configurable coroutine attributes, including stack size and shared stack address
* pfn :function the coroutine calls
* arg :argument of the function the coroutine calls
* @return :0
*/
int co_create( stCoRoutine_t **ppco,const stCoRoutineAttr_t *attr,pfn_co_routine_t pfn,void *arg )
{
if( !co_get_curr_thread_env() )
{
co_init_curr_thread_env(); //Initialize the environment of this thread. Only the main coroutine calls this
}
stCoRoutine_t *co = co_create_env( co_get_curr_thread_env(), attr, pfn,arg ); //Create the coroutine runtime environment and initialize the coroutine data
*ppco = co;
return 0;
}
co_create does two things. First, if the current thread has not initialized its runtime environment stCoRoutineEnv_t, it initializes it. This includes initializing the coroutine call stack, creating the main coroutine, and pushing it onto the stack. Second, it creates a coroutine from the attr parameter, allocates a private stack (or sets up the shared stack), and returns the handle co:
/* Coroutine switch-in interface
* @param
* co :pointer to the main coroutine structure
*/
void co_resume( stCoRoutine_t *co )
{
stCoRoutineEnv_t *env = co->env;
stCoRoutine_t *lpCurrRoutine = env->pCallStack[ env->iCallStackSize - 1 ]; //The coroutine that is currently running
if( !co->cStart ) //First time entering
{
coctx_make( &co->ctx,(coctx_pfn_t)CoRoutineFunc,co,0 ); //Save the context (current registers) in co->ctx
co->cStart = 1; //Mark as started
}
env->pCallStack[ env->iCallStackSize++ ] = co; //Push onto the coroutine call stack
co_swap( lpCurrRoutine, co ); //Switch
}
co_resume switches to a given coroutine. If co has not started yet, it initializes the coroutine stack with coctx_make. Then it pushes the coroutine onto the call stack and switches context with the current coroutine:
void co_yield_env( stCoRoutineEnv_t *env )
{
stCoRoutine_t *last = env->pCallStack[ env->iCallStackSize - 2 ];
stCoRoutine_t *curr = env->pCallStack[ env->iCallStackSize - 1 ];
env->iCallStackSize--;
co_swap( curr, last);
}
/* Interface to switch out the current coroutine
*/
void co_yield_ct()
{
co_yield_env( co_get_curr_thread_env() );
}
/* Coroutine switch-out interface
* @param
* co :pointer to the main coroutine structure
*/
void co_yield( stCoRoutine_t *co )
{
co_yield_env( co->env );
}
The co_yield family of functions makes the current coroutine give up the CPU. They pop it off the call stack and switch context with the previous coroutine on the stack.
//example_test.cpp
#include <stdio.h>
#include <stdlib.h>
#include "co_routine.h"
void* f(void* args) {
while (1) {
printf("f\n");
co_yield_ct();
}
return NULL;
}
void* g(void* args) {
while (1) {
printf("g\n");
co_yield_ct();
}
return NULL;
}
int main() {
stCoRoutine_t* co_f;
stCoRoutine_t* co_g;
co_create(&co_f, NULL, f, NULL);
co_create(&co_g, NULL, g, NULL);
while(1) {
co_resume(co_f);
co_resume(co_g);
}
return 0;
}
Using the three basic functions above, I wrote a small example here. The program creates two coroutines, f and g. Each coroutine prints its own function name and then gives up the CPU. The main coroutine calls co_resume in a loop to wake the two coroutines in turn. The program prints this in a loop:
./example_test
f
g
f
...
We have now covered the core functions of libco and how they execute. Thank you for reading. If you have any questions or thoughts, or find any mistake in this post, please let me know.
在之前的文章libco源码笔记(1)协程与上下文切换中，我们介绍了协程的基本概念以及libco中的上下文切换核心代码。本文结合一个示例，介绍libco提供的几个重要函数接口。建议配合我自己的注释版本阅读。
首先我们介绍libco中的三个核心结构体，下图1描述了三者的关系。
coctx_t保存协程切换时所需的上下文信息，详尽的说明请参考libco源码笔记(1)协程与上下文切换，此处不再赘述。
stCoRoutine_t协程主要结构体，包含单个协程的全部信息，如协程启停状态，执行函数，上下文信息，共享栈信息等。
stCoRoutineEnv_tstatic __thread stCoRoutineEnv_t* gCoEnvPerThread = NULL; //协程运行环境 __thread:线程私有
线程私有全局静态变量，包含全局协程环境信息，如协程调用栈，epoll句柄等。其中pCallStack为当前线程中的协程调用栈，由于libco为非对称协程
图1. libco核心结构
/* 协程创建接口
* @param
* co :协程主结构体二级指针
* attr :协程可配置属性, 包括栈大小、共享栈地址
* pfn :协程调用函数
* arg :协程调用函数参数
* @return :0
*/
int co_create( stCoRoutine_t **ppco,const stCoRoutineAttr_t *attr,pfn_co_routine_t pfn,void *arg )
{
if( !co_get_curr_thread_env() )
{
co_init_curr_thread_env(); //初始化本线程环境，主协程才会调用
}
stCoRoutine_t *co = co_create_env( co_get_curr_thread_env(), attr, pfn,arg ); //创建协程运行环境, 初始化协程数据
*ppco = co;
return 0;
}
co_create主要负责两件事情。首先，在当前线程没有初始化运行环境stCoRoutineEnv_t时，对其进行初始化，包括初始化协程调用栈，创建主协程并压栈等。其次，根据传入的配置参数attr创建协程，分配私有栈（设置共享栈）并返回句柄co。
/* 协程切回接口
* @param
* co :协程主结构体指针
*/
void co_resume( stCoRoutine_t *co )
{
stCoRoutineEnv_t *env = co->env;
stCoRoutine_t *lpCurrRoutine = env->pCallStack[ env->iCallStackSize - 1 ]; //当前正在运行的协程
if( !co->cStart ) //第一次进入
{
coctx_make( &co->ctx,(coctx_pfn_t)CoRoutineFunc,co,0 ); //在co->ctx中保存上下文(当前寄存器)
co->cStart = 1; //标记为已开始
}
env->pCallStack[ env->iCallStackSize++ ] = co; //压入协程调用栈
co_swap( lpCurrRoutine, co ); //切换
}
co_resume负责切换至某一协程。在co尚未启动时，通过coctx_make初始化协程栈，然后将协程压栈，并与当前协程进行上下文切换。
void co_yield_env( stCoRoutineEnv_t *env )
{
stCoRoutine_t *last = env->pCallStack[ env->iCallStackSize - 2 ];
stCoRoutine_t *curr = env->pCallStack[ env->iCallStackSize - 1 ];
env->iCallStackSize--;
co_swap( curr, last);
}
/* 当前协程切出接口
*/
void co_yield_ct()
{
co_yield_env( co_get_curr_thread_env() );
}
/* 协程切出接口
* @param
* co :协程主结构体指针
*/
void co_yield( stCoRoutine_t *co )
{
co_yield_env( co->env );
}
co_yield系列函数负责让当前协程让出CPU，将其出栈，并与栈上的前一个协程进行上下文切换。
//example_test.cpp
#include <stdio.h>
#include <stdlib.h>
#include "co_routine.h"
void* f(void* args) {
while (1) {
printf("f\n");
co_yield_ct();
}
return NULL;
}
void* g(void* args) {
while (1) {
printf("g\n");
co_yield_ct();
}
return NULL;
}
int main() {
stCoRoutine_t* co_f;
stCoRoutine_t* co_g;
co_create(&co_f, NULL, f, NULL);
co_create(&co_g, NULL, g, NULL);
while(1) {
co_resume(co_f);
co_resume(co_g);
}
return 0;
}
使用上面提到的三个基本函数，我在这里写了一个小例子，程序创建了两个协程f和g，每个协程在打印自己的函数名后，让出CPU。主协程循环调用co_resume依次唤起两个协程，程序的运行输出为循环打印：
./example_test
f
g
f
...
至此，我们了解了libco核心部分的相关函数与执行过程。感谢你的阅读。如果你有任何疑问和感想，或发现本文有任何错误，请一定让我知道。
This post uses WeChat’s high-performance open-source coroutine library libco to summarize the problems coroutines address and how they solve them. libco has few source comments, so here is my own annotated version. I suggest reading this post together with it. All code and explanations here run on x86 32-bit. The 64-bit case differs slightly. I skip it for space.
Before we start, let’s briefly recall why coroutines exist.
At first, as in Figure 1(a) below, our system has a steady stream of tasks (task in the figure) to process. We write a server program for it. The program runs as a single process (process in the figure) and keeps fetching tasks (loop in the figure). For each task it fetches, it calls the handler f() to run the actual logic. In f(), the code segment g() takes a long time. Even so, tasks arrive less often than f() takes to run. The system consumes tasks faster than they are produced, so the service runs fine.
As the business grows, we receive more tasks per unit of time. The single-process model in (a) can no longer consume tasks in time. So, as in Figure 1(b), we can split g() out into its own process, since it is fairly independent and uses many resources. The original process calls g() with an asynchronous remote call call_g(). It registers a callback g_callback() to handle the result of g(). When coding, we have to change from sequential programming to programming with a calling part and a callback part.
Figure 1
The asynchronous style raises system throughput and lowers coupling. But as the figure below shows, one sequential piece of code is split into several pieces. When the code is complex and needs many remote calls, maintainability drops sharply. We call this callback hell.
Figure 2. Code segments under synchronous and asynchronous programming
How do we solve callback hell? How do we keep execution asynchronous, yet turn the broken-up code back into the sequential flow we know? When C/C++ code runs, almost all of the runtime state is held in stack frames and registers. Suppose a remote call blocks. If we save the execution context ourselves, give up the CPU, and load the context again when the remote call returns, we can finish the asynchronous process inside one function stack. We call this mechanism a coroutine. It is like the process/thread switching we know, but the user triggers the context switch and manages it. So it is also often called a “user-space thread”.
Figure 3. Responsibilities of a coroutine library
What exactly is in the runtime “context” that we must save and load by hand? Take the following main function calling the sum function as an example:
int sum(int x, int y) {
int z = x + y;
return z;
}
int main() {
int a = 1;
int b = 10;
int c = sum(a, b);
return 0;
}
After compiling with g++ -m32 -S sum.cpp, the assembly is as follows:
_Z3sumii:
pushl %ebp
movl %esp, %ebp
subl $16, %esp ;make space for stack
movl 12(%ebp), %eax ;%eax = y
movl 8(%ebp), %edx ;%edx = x
addl %edx, %eax ;%eax = %eax + %edx
movl %eax, -4(%ebp) ;z = %eax
movl -4(%ebp), %eax ;%eax = z
leave ;%esp = %ebp; pop %ebp
ret ;pop %eip; jump(%eip)
main:
pushl %ebp
movl %esp, %ebp
subl $24, %esp ;make space for stack
movl $1, -4(%ebp) ;int a = 1;
movl $10, -8(%ebp) ;int b = 10;
movl -8(%ebp), %eax
movl %eax, 4(%esp) ;y = b;
movl -4(%ebp), %eax
movl %eax, (%esp) ;x = a;
call _Z3sumii ;push(eip); jump(sum);
movl %eax, -12(%ebp)
movl $0, %eax ;return 0;
As the figure below shows, the code mainly works on the stack frames of the two functions, shown in yellow and green. The base pointer register ebp and the stack pointer register esp mark the bottom and the top of the stack.
$L14$ first pushes the current ebp. Since main is the function that runs right after the process starts, ebp is 0 here.
$L15$ sets the ebp position for main.
$L16$ moves the esp address down by 16. This makes enough room for the local variables and the arguments of the sum call.
$L17-L22$ assign values to the variables a,b and to the arguments x,y of sum.
$L23$ runs the call instruction. It pushes the current instruction register eip and jumps to sum (eip points to the first instruction of sum).
$L2$ pushes the current ebp, which is ebp_main in the figure.
$L3$ sets the ebp position for sum, pointing at the current esp.
$L4$ makes room on the stack.
$L5, L6$ use ebp_sum + 8, ebp_sum + 12 to get the values of the arguments x, y.
$L7-L9$ do the addition and put the result in eax.
$L10$ calls the leave instruction. It moves esp back to the ebp position, pops ebp_main, and assigns it to ebp.
$L11$ calls the ret instruction. It pops the instruction address eip saved before the call to sum and assigns it to eip. Now the stack of main, the yellow part in the figure, is restored.
Line $L24$ assigns the result of sum in eax to c.
Line $L25$ assigns the return value 0 to eax. This finishes the whole process.
Figure 4. Function stack of sum.cpp
From this analysis, we can see that for a running function, arguments, return address, function stack, and registers make up all of its runtime information. With them, we can restore the execution state of any function. We call this the coroutine context.
coctx_t Context Informationlibco describes the coroutine context with the structure coctx_t defined below. ss_sp and ss_size hold the arguments, the return address, and the function stack. These are the red-box part of Figure 4. regs holds the registers for 32-bit/64-bit:
struct coctx_t
{
#if defined(__i386__)
void *regs[ 8 ]; //see coctx.cpp for details
#else
void *regs[ 14 ]; //see coctx.cpp for details. R10, R11 are callee saved registers, saved by the called function
#endif
size_t ss_size; //remaining size of the coroutine stack
char *ss_sp; //bottom address of the coroutine stack
};
co_make Context Initializationlibco uses the coctx_make below to fill in the initial content of the coroutine context on the first call (co_resume):
/*
* @param
* ctx :pointer to the context struct
* pfn :pointer to the function to call
* s :argument
* s1 :argument
*/
int coctx_make(coctx_t* ctx, coctx_pfn_t pfn, const void* s, const void* s1) {
// make room for coctx_param
char* sp = ctx->ss_sp + ctx->ss_size - sizeof(coctx_param_t); //ss_sp is heap memory, move sp to the high address
sp = (char*)((unsigned long)sp & -16L); //i386 requires the stack start address to be 16-byte aligned
coctx_param_t* param = (coctx_param_t*)sp;
void** ret_addr = (void**)(sp - sizeof(void*) * 2); //return address
*ret_addr = (void*)pfn;
param->s1 = s;
param->s2 = s1;
memset(ctx->regs, 0, sizeof(ctx->regs));
ctx->regs[kESP] = (char*)(sp) - sizeof(void*) * 2;
return 0;
}
Figure 5 below shows the coroutine stack after co_make fills it. It differs from the function call stack above. Before the arguments and the return address, 4 bytes are left empty (NULL in the figure). This prepares for the later context switch.
Figure 5. co_make initializes the coroutine stack
coctx_swap Context Switchingextern "C"
{
extern void coctx_swap( coctx_t *,coctx_t* ) asm("coctx_swap");
};
libco switches coroutine contexts with the coctx_swap function. It takes two coctx_t * arguments. The first points to where the current coroutine context is saved. The second points to the context to switch in:
.globl coctx_swap
coctx_swap:
movl 4(%esp), %eax ;eax = *(esp+4) get the first argument coctx_t
movl %esp, 28(%eax) ;coctx_t.regs[7] = esp
movl %ebp, 24(%eax) ;coctx_t.regs[6] = ebp
movl %esi, 20(%eax) ;coctx_t.regs[5] = esi
movl %edi, 16(%eax) ;coctx_t.regs[4] = edi
movl %edx, 12(%eax) ;coctx_t.regs[3] = edx
movl %ecx, 8(%eax) ;coctx_t.regs[2] = ecx
movl %ebx, 4(%eax) ;coctx_t.regs[1] = ebx
movl 8(%esp), %eax ;eax = *(esp+8) get the second argument coctx_t
movl 4(%eax), %ebx ;ebx = coctx_t.regs[1]
movl 8(%eax), %ecx ;ecx = coctx_t.regs[2]
movl 12(%eax), %edx ;edx = coctx_t.regs[3]
movl 16(%eax), %edi ;edi = coctx_t.regs[4]
movl 20(%eax), %esi ;esi = coctx_t.regs[5]
movl 24(%eax), %ebp ;ebp = coctx_t.regs[6]
movl 28(%eax), %esp ;esp = coctx_t.regs[7]
ret
Use the figure below as a reference. The stack before calling coctx_swap is the green part. The call instruction pushes the return address.
At $L3$, esp is at the position shown in the figure, and we enter the coctx_swap function.
$L3-L10$ save the register values into the coctx_t pointed to by the first argument.
$L12-L19$ read the information in the coctx_t pointed to by the second argument into the registers. This restores the context.
The ret instruction at $L21$ pops eip, the entry of the function pfn, and jumps to pfn. Now the yellow stack is built as the stack space before calling pfn (compare with the yellow part in the red box of Figure 4). The NULL reserved earlier is exactly what the ret instruction needs here.
Figure 6. coctx_swap context switching
Above, we saw how two coroutines switch contexts. For scheduling the coroutines, there are two main ways, as the figure below shows: symmetric coroutines and asymmetric coroutines.
In symmetric coroutines, all coroutines run as equals. They call transfer to jump freely between each other.
Asymmetric coroutines work like a coroutine call stack. At the start, only the main coroutine is on the stack. Calling resume pushes another coroutine onto the stack and switches to its context. When a coroutine finishes, or calls yield explicitly, it is popped and we switch back to the previous coroutine context. Usually the coroutine call stack is not deep. In practice, most switching is between the main coroutine (IO) and logic coroutines.
In real applications, symmetric coroutines cost more to maintain, and it is hard to keep track of the call chain. So asymmetric coroutines are more common. The libco described here is an asymmetric coroutine library.
Figure 7. Symmetric/asymmetric coroutines
From the coctx_make code above, we can see that the coroutine stack in libco is about ss_size in size. By default, when you call co_create to create a new coroutine in libco, it allocates ss_size of 128K on the heap and points ss_sp at it. Each coroutine then has its own stack space. This is called the “private stack” mode, also called stackful mode. In private stack mode, a context switch only needs to save and load registers, so the cost is low. But each coroutine’s stack size is fixed, so a lot of stack space is wasted.
In contrast to the private stack, libco offers a shared stack mode, also called stackless mode. In a shared stack, all coroutines use one fixed-size block of stack space (128K by default in libco). When a coroutine switches out, libco uses malloc to get a block of memory sized to the stack currently in use, and copies the shared stack content out to it. This uses memory more sensibly, but a context switch costs more.
So far, we have used the libco source to cover context switching, the core part of coroutines. Thank you for reading. If you have any questions or thoughts, or find any mistake in this post, please let me know.