[{"data":1,"prerenderedAt":1231},["ShallowReactive",2],{"content:\u002Fen\u002Fnotes\u002F65840\u002Freftextendedpaper":3,"series:content_en":520},{"id":4,"title":5,"authorship":6,"body":7,"categories":492,"date":494,"description":495,"draft":496,"extension":497,"image":498,"meta":499,"navigation":501,"path":502,"permalink":503,"published":503,"readingTime":504,"recommend":503,"references":503,"seo":509,"seoDescription":510,"seoTitle":503,"sitemap":511,"stem":512,"tags":513,"type":518,"__hash__":519},"content_en\u002Fposts\u002Fnotes\u002F65840\u002Freftextendedpaper.md","Reading the Raft Paper","human-only",{"type":8,"value":9,"toc":471},"minimark",[10,15,19,35,38,42,45,48,51,55,58,71,76,79,90,93,96,99,102,106,109,112,124,127,131,134,137,140,143,151,154,157,160,166,170,173,176,179,182,186,189,192,203,206,226,229,240,243,247,250,253,261,264,267,278,281,304,307,310,314,317,325,328,336,340,343,347,353,357,363,366,383,387,393,395,402,406,409,422,425,437,440,454,457],[11,12,14],"h3",{"id":13},"introduction","Introduction",[16,17,18],"p",{},"Raft is a consensus algorithm for managing log replication. Consensus algorithms are used in clusters of multiple machines to keep services running even when some of those machines fail. That makes them important for building reliable large-scale software systems.",[16,20,21,22,26,27,34],{},"The main Raft paper is ",[23,24,25],"em",{},"In Search of an Understandable Consensus Algorithm (Extended Version)",", available ",[28,29,33],"a",{"href":30,"rel":31},"https:\u002F\u002Fraft.github.io\u002Fraft.pdf",[32],"nofollow","here",". It's not very long, either: 18 pages. It compares Raft with Paxos throughout, opening with a fair amount of frustration, and emphasizes Raft's greatest strength: it's more understandable.",[16,36,37],{},"These are my notes from reading the paper.",[11,39,41],{"id":40},"background","Background",[16,43,44],{},"Consensus algorithms were developed primarily for the replicated state machine model. Replicated state machines are usually implemented with replicated logs, each containing a sequence of commands. Every machine in the cluster executes those commands in the same order and eventually reaches the same state. Notice the word “eventually”: this indicates eventual rather than strong consistency.",[16,46,47],{},"The consensus algorithm keeps the replicated logs consistent across the cluster. Each consensus module communicates with those on other machines to ensure that all machines eventually execute the same commands in the same order, even if some machines fail. Together, the machines can then provide a service as though they were a single machine.",[16,49,50],{},"This discussion of consensus applies only to non-Byzantine failures: nodes do not deliberately fabricate information.",[11,52,54],{"id":53},"the-algorithm","The Algorithm",[16,56,57],{},"Raft can be divided into three relatively independent parts:",[59,60,61,65,68],"ul",{},[62,63,64],"li",{},"Leader election: if the current leader fails, a new one must be elected.",[62,66,67],{},"Log replication: the leader receives log entries from clients and replicates them across the cluster, keeping other machines' logs consistent with its own.",[62,69,70],{},"Safety: once a machine accepts a particular command, another machine must not accept a different command at that same log index.",[72,73,75],"h4",{"id":74},"raft-basics","Raft Basics",[16,77,78],{},"A machine in a Raft cluster is always in one of three states:",[59,80,81,84,87],{},[62,82,83],{},"Leader: handles all client requests.",[62,85,86],{},"Follower: does not handle client requests; passively receives and responds to requests from leaders and candidates.",[62,88,89],{},"Candidate: participates in electing a leader.",[16,91,92],{},"During normal operation, the cluster has one leader and all other machines are followers.",[16,94,95],{},"Raft divides time into terms. Each term begins with an election in which one or more candidates try to become leader. If a candidate wins, it becomes the leader and the remaining machines become followers.",[16,97,98],{},"A term is a monotonically increasing integer. Each machine stores its current term and includes it in communications with other machines. If a machine discovers that its current term is lower than another machine's, it updates its own term. A candidate or leader that discovers a higher term, indicating a newer term, immediately becomes a follower.",[16,100,101],{},"Basic communication between Raft machines requires only two RPC types. RequestVote RPCs are initiated by candidates during elections. The Chinese rendering, literally “canvassing for votes,” sounds a little odd to me. AppendEntries RPCs are initiated by the leader to replicate logs and maintain heartbeats.",[72,103,105],{"id":104},"leader-election","Leader Election",[16,107,108],{},"A machine starts as a follower and remains one as long as it receives appropriate RPCs. The leader periodically sends AppendEntries requests containing no commands as heartbeats to maintain its role. If a follower receives no such request for a while, it starts an election.",[16,110,111],{},"The follower becomes a candidate, increments its current term, votes for itself, and sends RequestVote RPCs to the other machines in parallel. It remains a candidate until one of three things happens:",[113,114,115,118,121],"ol",{},[62,116,117],{},"A candidate receives votes from a majority of the cluster and wins. Each machine votes for at most one candidate in a term, on a first-come, first-served basis. This guarantees at most one winner per term. The winner becomes leader and sends heartbeats to the other machines to establish its role.",[62,119,120],{},"The candidate receives an AppendEntries request from another machine. If the request's term is greater than or equal to the candidate's current term, it becomes a follower. Otherwise, it rejects the request and remains a candidate.",[62,122,123],{},"Several followers become candidates at the same time, and none receives a majority. The candidates time out, increment their current terms, and start another election.",[16,125,126],{},"To keep the third case from repeating forever, election timeouts are randomized within a fixed interval, reducing the chance of simultaneous timeouts.",[72,128,130],{"id":129},"log-replication","Log Replication",[16,132,133],{},"Each client request contains a command for the replicated state machine to execute. The leader appends the command to its log and sends AppendEntries requests to other machines in parallel to replicate it. Once replication completes, the leader commits the command to the state machine and replies to the client with success.",[16,135,136],{},"Besides the command to replicate, an AppendEntries request carries the term in which the leader received the command and an integer index identifying its position in the log.",[16,138,139],{},"The leader commits a command once it has successfully replicated it to a majority of machines. Raft guarantees that committed commands are durable and will eventually be executed by every available state machine. Committing an entry also commits all preceding entries, including those created by earlier leaders. The leader tracks the commit index and includes it in every AppendEntries request. When a follower learns that an entry has been committed, it applies that entry to its state machine as well.",[16,141,142],{},"Raft also needs to guarantee two properties:",[59,144,145,148],{},[62,146,147],{},"If two log entries have the same term and log index, they contain the same command.",[62,149,150],{},"If two log entries have the same term and log index, all entries preceding them are identical as well.",[16,152,153],{},"The first is fairly straightforward; the second needs more attention.",[16,155,156],{},"In each AppendEntries request, the leader includes the term and index of the entry immediately preceding the new entries. If the follower cannot find a matching entry in its log, it rejects the request. This is the consistency check. A successful response therefore tells the leader that the follower's log agrees with its own.",[16,158,159],{},"When the consistency check fails, the leader forces the follower's log to match its own. It finds their last matching entry, removes all later entries from the follower's log, and sends its own subsequent entries. The leader maintains a nextIndex for each follower, recording the index of the next entry to send. A newly elected leader initializes every nextIndex to one past the last index in its own log. When a follower fails the check, the leader decrements that follower's nextIndex and retries AppendEntries.",[161,162,163],"blockquote",{},[16,164,165],{},"This part feels a little vague. I assume each AppendEntries request sends entries from nextIndex through the last entry. Otherwise, passing the consistency check would only trim the follower's log back to the point of agreement, without copying the leader's later entries to it.",[72,167,169],{"id":168},"safety","Safety",[16,171,172],{},"The mechanism described so far does not completely guarantee safety. For example, a machine could become unreachable while the leader commits several entries, then later be elected leader and overwrite those entries. Raft adds an election restriction to prevent this. It ensures that a leader in any term contains all entries committed in preceding terms.",[16,174,175],{},"First, RequestVote includes information about the candidate's log. If a machine finds that its own log is more up to date than the candidate's, by comparing the last entry's term and index, it refuses the vote.",[16,177,178],{},"A leader commits an entry from its current term once a majority of machines has accepted it. If the leader fails while committing the entry, the next leader continues trying to replicate it. However, the new leader cannot immediately conclude that an entry from an earlier term is committed merely because it is stored on a majority of machines. This can lead to the problem shown in Figure 8 of the paper.",[16,180,181],{},"Raft therefore does not commit entries from earlier terms simply by counting replicas. It uses replica counts only to commit entries from the current term. Once an entry from the current term is committed, all preceding entries are committed implicitly.",[72,183,185],{"id":184},"cluster-membership-changes","Cluster Membership Changes",[16,187,188],{},"Changing cluster membership without taking the entire cluster offline can create two independent majorities, potentially allowing two leaders to be elected.",[16,190,191],{},"Raft uses a two-stage approach to make membership changes safe. The cluster first enters a joint consensus state. Once that configuration is committed, it switches to the new configuration. The cluster can continue serving requests during joint consensus. In this state:",[59,193,194,197,200],{},[62,195,196],{},"Log entries are replicated to every machine in both the old and new configurations.",[62,198,199],{},"Any machine in either configuration may become leader.",[62,201,202],{},"Elections and log replication require separate majorities of both the old and new configurations.",[16,204,205],{},"Configurations are stored and transmitted as special log entries. The process is:",[113,207,208,211,214,217,220,223],{},[62,209,210],{},"The leader receives a request to change the configuration from C_old to C_new.",[62,212,213],{},"It stores C_old and C_new together in a log entry as the joint configuration C_old,new.",[62,215,216],{},"It appends that entry to machines in both configurations.",[62,218,219],{},"As soon as a machine adds the entry to its log, even before commitment, it uses that configuration for subsequent operations.",[62,221,222],{},"Once C_old,new has been accepted by the required majorities, the leader commits it. At this point, a leader can no longer be elected under C_old or C_new alone.",[62,224,225],{},"The leader creates a C_new log entry, replicates it to the machines, and commits it.",[16,227,228],{},"Three issues remain:",[113,230,231,234,237],{},[62,232,233],{},"A new machine has no log and needs time to catch up, which could temporarily reduce availability. Raft adds a preliminary stage in which new machines receive AppendEntries but are not voting members, so consensus does not depend on them. Once they catch up, the configuration change proceeds as described above.",[62,235,236],{},"The leader may not belong to the new configuration. In that case, it steps down when it commits C_new. For a while, then, it manages a cluster that does not include itself: it replicates entries but does not count itself toward a majority.",[62,238,239],{},"Removed servers can disrupt the cluster. Since they no longer receive heartbeats, they may start elections and send RequestVote RPCs with higher terms, causing the current leader to become a follower. These elections cannot succeed, and a new leader will still come from the new cluster, but the removed machines can repeatedly time out and harm availability.",[16,241,242],{},"To prevent the third issue, Raft adds a restriction: a server that receives RequestVote before the election timeout has elapsed since its last heartbeat from the current leader does not update its term or grant a vote. As long as the leader maintains heartbeats with the current cluster, a vote request with a higher term cannot unseat it.",[72,244,246],{"id":245},"log-compaction","Log Compaction",[16,248,249],{},"As the log grows, machines cannot keep all of it in memory. Snapshots periodically save the system state to persistent storage, allowing log entries up to the snapshot point to be safely removed from memory.",[16,251,252],{},"Each machine manages its own snapshots independently. A snapshot includes only committed entries. Besides the current state of the state machine, it stores two pieces of metadata:",[59,254,255,258],{},[62,256,257],{},"The index of the last entry included in the snapshot.",[62,259,260],{},"The term of that entry.",[16,262,263],{},"This metadata is mainly needed for the AppendEntries consistency check, which compares the preceding log entry. To support membership changes, the snapshot must also contain the latest configuration as of its snapshot point. Once the snapshot has been written, the machine can delete the covered log entries and older snapshots.",[16,265,266],{},"Sometimes a leader needs to send a snapshot to a newly joined or lagging node. It uses a new RPC, InstallSnapshot, to do so:",[268,269,275],"pre",{"className":270,"code":272,"language":273,"meta":274},[271],"language-go","type InstallSnapshotRequest struct {\n    \u002F\u002F Term Leader 的任期\n    Term              int64\n    \u002F\u002F LeaderID Follower 可以将客户端请求重定向到 Leader\n    LeaderID          int64\n    \u002F\u002F LastIncludedIndex 快照包含的最后一个条目的索引\n    LastIncludedIndex int64\n    \u002F\u002F LastIncludedTerm 快照包含的最后一个条目的任期\n    LastIncludedTerm  int64\n    \u002F\u002F Offset 快照文件中的该快照块的偏移\n    Offset            int64\n    \u002F\u002F Data 快照块数据\n    Data              []byte\n    \u002F\u002F Done 是否是最后一个快照块\n    Done              bool\n}\n\ntype InstallSnapshotResponse struct {\n    \u002F\u002F Term Follower 当前任期\n    Term    int64\n}\n","go","",[276,277,272],"code",{"__ignoreMap":274},[16,279,280],{},"The receiver proceeds as follows:",[113,282,283,286,289,292,295,298,301],{},[62,284,285],{},"If Term is less than CurrentTerm, return immediately.",[62,287,288],{},"If this is the first snapshot chunk, Offset = 0, create the snapshot file.",[62,290,291],{},"Write the data at the specified offset.",[62,293,294],{},"If Done is false, return and wait for the next InstallSnapshot request.",[62,296,297],{},"If the log contains an entry matching LastIncludedIndex and LastIncludedTerm, retain all entries after it and return.",[62,299,300],{},"Discard the entire log.",[62,302,303],{},"Reset the state machine using the snapshot and adopt the configuration stored in it.",[16,305,306],{},"Usually, a snapshot covers entries that the receiver does not yet have. In that case, the receiver discards its log and uses the snapshot. If the receiver already has all the entries covered by the snapshot, the snapshot replaces that portion of the log, but subsequent entries must be retained.",[16,308,309],{},"There are also performance considerations. A machine can take a snapshot whenever its log reaches a fixed size in bytes. That threshold should be neither too large nor too small: too large makes snapshots slow to write, while too small makes them too frequent. Writing a snapshot can take a long time, since disk I\u002FO is slow, and interfere with normal processing. Raft recommends copy-on-write so the machine can continue appending entries and serving requests while writing the snapshot to disk.",[72,311,313],{"id":312},"client-interaction","Client Interaction",[16,315,316],{},"The Raft cluster's leader handles all client requests. A client initially sends its request to an arbitrary machine. If that machine is not the leader, it rejects the request and returns the address of the leader from which it most recently received a heartbeat. If the leader fails, the request times out and the client tries another arbitrary machine.",[16,318,319,320,324],{},"Raft aims to provide linearizable semantics: each operation appears to happen instantaneously and is executed only once. But a leader may execute a command and fail before replying. The client then sends the request again to another leader, potentially executing it twice. The solution is for the ",[321,322,323],"strong",{},"clients"," to assign monotonically increasing unique identifiers to their commands, while the state machine records the identifier of the last executed command. If it receives a command that has already run, it immediately returns success without executing it again.",[16,326,327],{},"Read-only requests need not write to the log and can therefore be handled without reaching consensus for each request. But the supposed leader may already have been replaced in a newer term without knowing it, in which case a read could return stale data. Raft prevents this with two measures:",[113,329,330,333],{},[62,331,332],{},"The leader must have up-to-date information about all committed entries. Leader completeness guarantees it has those entries, but at the beginning of its term it may not know which are committed. Raft addresses this by committing a no-op entry at the start of the term to establish the current commit information.",[62,334,335],{},"Before serving a read-only request, the leader must establish that it is still the leader. It can do so by exchanging heartbeats with a majority of the cluster before handling the request.",[11,337,339],{"id":338},"implementing-the-algorithm","Implementing the Algorithm",[16,341,342],{},"Figure 2 of the paper gives a very detailed implementation outline. That's why Raft is awesome! It excludes membership changes and log compaction.",[72,344,346],{"id":345},"server-state","Server State",[268,348,351],{"className":349,"code":350,"language":273,"meta":274},[271],"type ServerState struct {\n    \u002F***** 所有 Server 都包含的持久状态 *****\u002F\n    \u002F\u002F CurrentTerm 机器遇到的最大的任期，启动时初始化为 0，单调递增\n    CurrentTerm int64;\n    \u002F\u002F VotedFor 当前任期内投票的 Candidate ID，未投票则为 nil\n    VotedFor    *int64;\n    \u002F\u002F Logs 日志条目，每个条目都包含了一条状态机指令和 Leader 接收该条目时的任期，index 从 1 开始\n    Logs        []*Log;\n\n    \u002F***** 所有 Server 都包含的可变状态 *****\u002F\n    \u002F\u002F CommitIndex 已知的最大的即将提交的日志索引，启动时初始化为 0，单调递增\n    CommitIndex int64;\n    \u002F\u002F LastApplied 最大的已提交的日志索引，启动时初始化为 0，单调递增\n    LastApplied int64;\n\n    \u002F******* Leader 包含的可变状态，选举后初始化 *******\u002F\n    \u002F\u002F NextIndex 每台机器下一个要发送的日志条目的索引，初始化为 Leader 最后一个日志索引 +1\n    NextIndex  []int64;\n    \u002F\u002F MatchIndex 每台机器已知复制的最高的日志条目，初始化为 0，单调递增\n    MatchIndex []int64;\n}\n",[276,352,350],{"__ignoreMap":274},[72,354,356],{"id":355},"appendentries","AppendEntries",[268,358,361],{"className":359,"code":360,"language":273,"meta":274},[271],"type AppendEntriesRequest struct {\n    \u002F\u002F Term Leader 的任期\n    Term         int64\n    \u002F\u002F LeaderID Follower 可以将客户端请求重定向到 Leader\n    LeaderID     int64\n    \u002F\u002F PrevLogIndex 新日志条目前一个日志条目的日志索引\n    PrevLogIndex int64\n    \u002F\u002F PrevLogTerm 前一个日志条目的任期\n    PrevLogTerm  int64\n    \u002F\u002F Entries 需要保存的日志条目，心跳包为空\n    Entries      []*Log\n    \u002F\u002F LeaderCommit Leader 的 CommitIndex\n    LeaderCommit int64\n}\n \ntype AppendEntriesResponse struct {\n    \u002F\u002F Term Follower 当前任期\n    Term    int64\n    \u002F\u002F Success Follower 包含 PrevLogIndex 和 PrevLogTerm 的日志条目为 true\n    Success bool\n}\n",[276,362,360],{"__ignoreMap":274},[16,364,365],{},"The receiver's rules:",[113,367,368,371,374,377,380],{},[62,369,370],{},"Return false if Term is less than CurrentTerm.",[62,372,373],{},"Return false if the log has no entry matching PrevLogIndex and PrevLogTerm.",[62,375,376],{},"If an existing entry has the same index as a new entry but a different term, delete that entry and everything after it.",[62,378,379],{},"Append any entries not already present in the log.",[62,381,382],{},"If LeaderCommit is greater than CommitIndex, set CommitIndex to the smaller of LeaderCommit and the index of the last new entry.",[72,384,386],{"id":385},"requestvote","RequestVote",[268,388,391],{"className":389,"code":390,"language":273,"meta":274},[271],"type RequestVoteRequest struct {\n    \u002F\u002F Term Candidate 的任期\n    Term         int64\n    \u002F\u002F CandidateId 拉票的 Candidate 的 ID\n    CandidateId  int64\n    \u002F\u002F LastLogIndex Candidate 最后一条日志序列的索引\n    LastLogIndex int64\n    \u002F\u002F LastLogTerm Candidate 最后一条日志序列的任期\n    LastLogTerm  int64\n}\n\ntype RequestVoteResponse struct {\n    \u002F\u002F Term 当前任期\n    Term        int64\n    \u002F\u002F VoteGranted true 则拉票成功\n    VoteGranted bool\n}\n",[276,392,390],{"__ignoreMap":274},[16,394,365],{},[113,396,397,399],{},[62,398,370],{},[62,400,401],{},"Return true if (VotedFor is nil or CandidateId) and the candidate's log is at least as up to date as the receiver's.",[72,403,405],{"id":404},"server-rules","Server Rules",[16,407,408],{},"For all machines:",[59,410,411,419],{},[62,412,413,414,418],{},"If CommitIndex is greater than LastApplied, increment LastApplied and apply log",[415,416,417],"span",{},"LastApplied"," to the state machine.",[62,420,421],{},"If an RPC request or response contains a Term greater than CurrentTerm, update CurrentTerm and become a follower.",[16,423,424],{},"For followers:",[59,426,427,430],{},[62,428,429],{},"Respond to RPCs from candidates and leaders.",[62,431,432,433,436],{},"If the election timeout elapses without receiving AppendEntries from the current leader or granting a vote to a candidate, become a candidate. Note that it is ",[23,434,435],{},"granting a vote",", not merely receiving a vote request, that matters.",[16,438,439],{},"For candidates:",[59,441,442,445,448,451],{},[62,443,444],{},"On becoming a candidate, start an election: increment the current term, vote for yourself, reset the election timer, and send RequestVote to all other machines.",[62,446,447],{},"Become leader if a majority grants votes.",[62,449,450],{},"Become a follower if AppendEntries arrives from a new leader.",[62,452,453],{},"Start a new election if the election timeout elapses.",[16,455,456],{},"For leaders:",[59,458,459,462,465,468],{},[62,460,461],{},"On becoming leader, send empty AppendEntries to all other machines. Continue sending them when idle to prevent election timeouts.",[62,463,464],{},"On receiving a client command, append it to the log and respond after it has been applied to the state machine.",[62,466,467],{},"If the last log index is greater than a follower's NextIndex, send AppendEntries containing every entry from NextIndex onward. On success, update the follower's NextIndex and MatchIndex. On failure due to log inconsistency, decrement NextIndex and retry.",[62,469,470],{},"If there is an N greater than CommitIndex such that a majority of MatchIndex values are at least N and entry N is from the current term, set CommitIndex to N.",{"title":274,"searchDepth":472,"depth":472,"links":473},4,[474,476,477,486],{"id":13,"depth":475,"text":14},3,{"id":40,"depth":475,"text":41},{"id":53,"depth":475,"text":54,"children":478},[479,480,481,482,483,484,485],{"id":74,"depth":472,"text":75},{"id":104,"depth":472,"text":105},{"id":129,"depth":472,"text":130},{"id":168,"depth":472,"text":169},{"id":184,"depth":472,"text":185},{"id":245,"depth":472,"text":246},{"id":312,"depth":472,"text":313},{"id":338,"depth":475,"text":339,"children":487},[488,489,490,491],{"id":345,"depth":472,"text":346},{"id":355,"depth":472,"text":356},{"id":385,"depth":472,"text":386},{"id":404,"depth":472,"text":405},[493],"notes","2022-12-03 21:40:09","Raft is a consensus algorithm designed to improve the efficiency of log replication. It is particularly suited to clusters of machines, allowing them to keep providing service even when some machines fail. It uses the replicated state machine model: logs record the order of commands so that every machine in the cluster can reach the same state. In Search of an Understandable Consensus Algorithm explores Raft’s design and compares it with Paxos, highlighting its understandability and providing a foundation for building reliable large-scale software systems. These reading notes aim to help explain the paper’s core concepts and their applications.",false,"md","https:\u002F\u002Fblog-img.774352199.xyz\u002F7mmvIZ.webp",{"slots":500},{},true,"\u002Fnotes\u002F65840\u002Freftextendedpaper",null,{"text":505,"minutes":506,"time":507,"words":508},"16 min read",15.98,958800,3196,{"title":5,"description":495},"Notes on the extended Raft paper, covering leader election, log replication, commit safety, joint consensus, snapshots, and client request handling.",{"loc":502},"posts\u002Fnotes\u002F65840\u002Freftextendedpaper",[514,515,516,517],"Raft","Distributed consensus","Log replication","Paper notes","tech","B8CgdF8vwFVig_8LK5yVvrlO9XcwQahMoAOFMu756v8",[521,539,556,574,591,610,628,646,664,680,696,711,727,743,760,778,793,810,824,840,857,874,890,907,923,942,958,975,993,1007,1025,1041,1056,1070,1074,1091,1105,1120,1135,1149,1163,1177,1192,1206,1217],{"categories":522,"date":524,"description":525,"image":526,"path":527,"readingTime":528,"recommend":503,"tags":533,"title":538,"type":518},[523],"daily","2024-09-01 22:03:10","Chronic gastritis has been a long, uneven journey: stomach trouble throughout childhood, a surprising reprieve at university, then a return of symptoms after years of late nights and drinking. Frequent nausea and reflux eventually became too much to live with. After repeated examinations and some reflection, I finally began taking the recovery process seriously.","https:\u002F\u002Fblog-img.774352199.xyz\u002F0Hr9l5.webp","\u002Fdaily\u002Fanti-chronic-gastritis",{"text":529,"minutes":530,"time":531,"words":532},"4 min read",3.08,184800,616,[534,535,536,537],"Chronic gastritis","Medical care experiences","Gastroscopy","Health journal","My Journey with Chronic Gastritis",{"categories":540,"date":541,"description":542,"image":543,"path":544,"readingTime":545,"recommend":503,"tags":550,"title":555,"type":518},[523],"2022-04-11 00:13:13","In The Three-Body Problem, Liu Cixin uses an immense historical canvas to explore the relationship between individuals and the collective. His doubts about Western democracy and the majority’s ability to determine its own fate run through the trilogy. Heroes struggle and sacrifice, only to see their work swept aside by history. Beginning with the Cultural Revolution, the story sets up a conflict between exceptional individuals and the ordinary masses, raising uncomfortable questions about humanity and society.","https:\u002F\u002Fblog-img.774352199.xyz\u002FLTzbFP.webp","\u002Fdaily\u002Fpeople-in-three-body",{"text":546,"minutes":547,"time":548,"words":549},"9 min read",8.635,518100,1727,[551,552,553,554],"The Three-Body Problem","Liu Cixin","Science fiction","Reading notes","The “Majority” in The Three-Body Problem",{"categories":557,"date":558,"description":559,"image":560,"path":561,"readingTime":562,"recommend":503,"tags":567,"title":573,"type":518},[523],"2024-01-04 02:06:51","My learning goals for 2024: reach N2 in Japanese, finish SICP and TAPL, build a kernel with full POSIX support, and update the blog theme. A challenging list, but a clear direction for the year.","https:\u002F\u002Fblog-img.774352199.xyz\u002FV3rSQC.webp","\u002Fdaily\u002Fplan2024",{"text":563,"minutes":564,"time":565,"words":566},"1 min read",0.15,9000,30,[568,569,570,571,572],"Annual planning","Learning Japanese","SICP","TAPL","Operating systems","My Learning Plan for 2024",{"categories":575,"date":576,"description":577,"image":578,"path":579,"readingTime":580,"recommend":503,"tags":585,"title":590,"type":518},[523],"2024-09-28 16:26:00","Three years at ByteDance have made time feel strangely accelerated. Amid the changes, I have still managed to hold on to some mental balance. Looking back at the new graduate gazing at distant mountains from the balcony of our Hangzhou office, I can trace how curiosity, workplace challenges, and shifting expectations taught me to find a rhythm of my own.","https:\u002F\u002Fblog-img.774352199.xyz\u002FhnCaht.webp","\u002Fdaily\u002Fwork-for-3-years",{"text":581,"minutes":582,"time":583,"words":584},"10 min read",9.74,584400,1948,[586,587,588,589],"ByteDance","Software engineering careers","Career retrospective","Mental health","Working at ByteDance for Three Years and Staying Somewhat Sane Is Not Entirely Impossible",{"categories":592,"date":594,"description":595,"image":596,"path":597,"readingTime":598,"recommend":503,"tags":603,"title":609,"type":518},[593],"fiddling","2025-05-28 22:09:00","After moving my blog to Astro, the usual Google Analytics integration no longer fit its performance goals. Adding event-reporting JavaScript directly to the head works, but hurts page performance. I used partytown to move the scripts off the main thread so they would not interfere with loading. With a few adjustments to the example code, Google Analytics finally worked, balancing performance with analytics.","https:\u002F\u002Fblog-img.774352199.xyz\u002FQ0w4RN.webp","\u002Ffiddling\u002Fastro-google-tag-manager",{"text":599,"minutes":600,"time":601,"words":602},"2 min read",1.785,107100,357,[604,605,606,607,608],"Astro","Google Analytics","Google Tag Manager","Partytown","Web analytics","Adding Google Analytics to Astro with Tag Manager",{"categories":611,"date":612,"description":613,"image":614,"path":615,"readingTime":616,"recommend":621,"tags":622,"title":627,"type":518},[593],"2023-04-08 13:16:36","Designing a new programming language is challenging and fun. Setting aside complicated compiler theory and implementation details to focus on where code runs helps clarify how languages are built. Starting from the RISC-VI instruction set, this discussion explores the underlying architecture, layered computer systems, and virtual-machine model, reflecting on the nature of programming languages as well as their implementation.","https:\u002F\u002Fblog-img.774352199.xyz\u002FuO420B.webp","\u002Ffiddling\u002Fchitchat-about-programming-language",{"text":617,"minutes":618,"time":619,"words":620},"15 min read",14.275,856500,2855,2,[623,624,625,626],"Programming language design","Type systems","Compiler design","RISC-V","Some Thoughts on Programming Languages",{"categories":629,"date":630,"description":631,"image":632,"path":633,"readingTime":634,"recommend":472,"tags":639,"title":645,"type":518},[593],"2025-04-18 16:43:12","My girlfriend moved from Beijing to Shanghai for work, and I helped arrange broadband too. Shanghai Telecom’s 500M connection costs more than a 1000M line in Hangzhou, frustratingly. I set out to connect the two cities’ networks: transparent proxying in Shanghai, selected traffic exiting through Hangzhou, and access between both LANs. Hangzhou already had a simple setup with a software router and an AP, configured to route my everyday traffic home and ready for the next networking adventure.","https:\u002F\u002Fblog-img.774352199.xyz\u002FO6cAGh.webp","\u002Ffiddling\u002Fcross-city-network-setup",{"text":635,"minutes":636,"time":637,"words":638},"5 min read",4.865,291900,973,[640,641,642,643,644],"Tailscale","mihomo","Site-to-site networking","Transparent proxy","Home networking","Connecting My Shanghai and Hangzhou Networks",{"categories":647,"date":648,"description":649,"image":650,"path":651,"readingTime":652,"recommend":503,"tags":657,"title":663,"type":518},[593],"2021-12-27 00:09:00","The labs are an essential part of learning CSAPP, but setting up Linux can be discouraging. Virtual machines bring installation errors, compatibility problems, and broken networking. WSL (Windows Subsystem for Linux), especially on Windows 10 version 2004 and later, provides a simpler, more direct Linux environment without the complexity and performance bottlenecks of a traditional VM.","https:\u002F\u002Fblog-img.774352199.xyz\u002FvqOC7N.webp","\u002Ffiddling\u002Fcsapplab0",{"text":653,"minutes":654,"time":655,"words":656},"6 min read",5.295,317700,1059,[658,659,660,661,662],"CSAPP","WSL2","Linux","GDB","Lab environment","Setting Up the CSAPP Lab Environment",{"categories":665,"date":666,"description":667,"image":668,"path":669,"readingTime":670,"recommend":503,"tags":675,"title":679,"type":518},[593],"2024-07-13 17:49:00","Using Debian as a side router offers a more stable and flexible alternative without depending on OpenWrt and LuCI. Configuring Debian directly gives you greater control over the system and avoids the limitations and instability of a GUI. Compared with common side-router setups, this approach makes transparent proxying more reliable and offers another option for those who value performance and efficiency.","https:\u002F\u002Fblog-img.774352199.xyz\u002FpPRU5x.webp","\u002Ffiddling\u002Fdebian-as-bypass-router",{"text":671,"minutes":672,"time":673,"words":674},"12 min read",11.79,707400,2358,[676,677,641,678,643],"Debian","Side router","AdGuard Home","Using Debian as a Side Router",{"categories":681,"date":682,"description":683,"image":684,"path":685,"readingTime":686,"recommend":503,"tags":690,"title":695,"type":518},[593],"2024-08-16 23:53:00","This FakeIP-based transparent proxy design addresses the single point of failure, poor performance, and awkward port forwarding of a traditional side router. Switching to the sing-box proxy core improves forwarding performance and simplifies configuration, with broader protocol support and better optimization than the previous Clash setup. Clash can implement the same design, but sing-box provides a flexible alternative.","https:\u002F\u002Fblog-img.774352199.xyz\u002FS2HHD5.webp","\u002Ffiddling\u002Ffake-ip-based-transparent-proxy",{"text":581,"minutes":687,"time":688,"words":689},9.395,563700,1879,[691,692,643,693,694],"FakeIP","sing-box","Policy-based routing","Traffic routing","Routing Transparent Proxy Traffic with FakeIP",{"categories":697,"date":698,"description":699,"image":700,"path":701,"readingTime":702,"recommend":503,"tags":706,"title":710,"type":518},[593],"2024-08-15 23:50:00","Port forwarding on the main router often stops working when a side router is introduced. Setting the side router as the gateway changes the forwarding path, breaking mappings that previously relied on the main router. A gateway translates addresses and forwards traffic from the internal network to the outside, and each internal device needs one to communicate externally. Understanding this mechanism helps explain how to fix the forwarding problem.","https:\u002F\u002Fblog-img.774352199.xyz\u002FmRqws9.webp","\u002Ffiddling\u002Ffix-port-forward-in-bypass-router",{"text":529,"minutes":703,"time":704,"words":705},3.03,181800,606,[677,707,708,709],"Port forwarding","NAT","Network troubleshooting","Fixing Port Forwarding with a Side Router",{"categories":712,"date":713,"description":714,"image":715,"path":716,"readingTime":717,"recommend":503,"tags":722,"title":726,"type":518},[593],"2023-02-02 23:24:55","Inspiration struck during Chinese New Year: an article read on the train about running Go on bare metal sparked an interest in low-level system interfaces. Its successful implementation suggested exciting possibilities for combining a high-level language with an OS. Further research revealed earlier work on the idea, and that growing enthusiasm became a project full of anticipation that ultimately did not work out.","https:\u002F\u002Fblog-img.774352199.xyz\u002FxB1Ni5.webp","\u002Ffiddling\u002Fgo-os",{"text":718,"minutes":719,"time":720,"words":721},"8 min read",7.29,437400,1458,[723,626,572,724,725],"Go","Bare-metal programming","Runtime","A Project That Failed: Seven Not-So-Happy Days over Chinese New Year",{"categories":728,"date":729,"description":730,"image":731,"path":732,"readingTime":733,"recommend":503,"tags":738,"title":742,"type":518},[593],"2022-08-15 01:05:01","While refactoring a system, converting entities between layers made deep copying surprisingly awkward. A product VO in the view layer, an entity in the domain layer, and a PO in the persistence layer can look nearly identical, yet small type differences complicate direct conversion. I used reflection to build a general conversion method, reducing repetitive assembler methods and making the code more maintainable and flexible.","https:\u002F\u002Fblog-img.774352199.xyz\u002FBlhm0I.webp","\u002Ffiddling\u002Fgolang-deepcopy-between-different-type",{"text":734,"minutes":735,"time":736,"words":737},"3 min read",2.855,171300,571,[723,739,740,741],"Reflection","Deep copy","Struct conversion","Deep Copying Between Different Struct Types in Go",{"categories":744,"date":745,"description":746,"image":747,"path":748,"readingTime":749,"recommend":503,"tags":753,"title":759,"type":518},[593],"2025-03-31 23:51:00","Periodically syncing heart-rate data from Apple Health to a server and visualizing it in Grafana provides an intuitive way to monitor it. Health Auto Export sends the data to an HTTP endpoint through its REST API automation, the server stores it in InfluxDB, and Grafana presents a clear dashboard for tracking and analyzing personal heart-rate changes.","https:\u002F\u002Fblog-img.774352199.xyz\u002FF4qD2T.webp","\u002Ffiddling\u002Fheart-rate-to-grafana",{"text":734,"minutes":750,"time":751,"words":752},2.45,147000,490,[754,755,756,757,758],"Apple Watch","Health Auto Export","InfluxDB","Grafana","Heart rate monitoring","My Heart Beats for U: Visualizing Heart Rate in Grafana",{"categories":761,"date":762,"description":763,"image":764,"path":765,"readingTime":766,"recommend":770,"tags":771,"title":777,"type":518},[593],"2025-06-10 20:18:00","macOS and iPadOS 26 introduce the Liquid Glass design language, refreshing icons and windows with a more modern look. Windowed apps on iPad are a step toward making it a productivity device. The transparent Control Center and the integration of Launchpad have been controversial, however, and the experience still needs work. Despite its shortcomings, this update lays groundwork for a promising future.","https:\u002F\u002Fblog-img.774352199.xyz\u002FTPSaLE.webp","\u002Ffiddling\u002Fmacos-26-trial",{"text":529,"minutes":767,"time":768,"words":769},3.305,198300,661,5,[772,773,774,775,776],"macOS 26","iPadOS 26","Liquid Glass","Apple Intelligence","Operating system impressions","Trying Out macOS 26 and iPadOS 26",{"categories":779,"date":780,"description":781,"image":782,"path":783,"readingTime":784,"recommend":503,"tags":788,"title":792,"type":518},[593],"2026-06-11 10:00:00","Another WWDC brings new Apple Intelligence features in macOS 27. The workarounds from macOS 26 no longer get through, so the battle of wits with Apple continues.","https:\u002F\u002Fblog-img.774352199.xyz\u002FfOFucm.webp","\u002Ffiddling\u002Fmacos-27-apple-intelligence-chatgpt",{"text":653,"minutes":785,"time":786,"words":787},5.1,306000,1020,[789,775,790,791],"macOS 27","ChatGPT","Mainland China Mac","Getting Apple Intelligence Working on a China-Market Mac: Region Changes and Extracting ChatGPT",{"categories":794,"date":795,"description":796,"image":797,"path":798,"readingTime":799,"recommend":503,"tags":803,"title":809,"type":518},[593],"2025-07-20 23:32:00","MoonTV is a new video aggregation platform built with Next.js and React to make following shows convenient. It began as an attempt to improve LibreTV and has attracted substantial attention and usage over several months of development. Cursor made development efficient, although multi-platform support and complex data dependencies posed challenges. As its user base grows, MoonTV continues improving in response to feedback.","https:\u002F\u002Fblog-img.774352199.xyz\u002FnIeONi.webp","\u002Ffiddling\u002Fmoontv-vibe-coding",{"text":653,"minutes":800,"time":801,"words":802},5.76,345600,1152,[804,805,806,807,808],"MoonTV","LunaTV","Vibe Coding","Cursor","AI-assisted development","MoonTV: An Experiment in Vibe Coding",{"categories":811,"date":812,"description":813,"image":814,"path":815,"readingTime":816,"recommend":503,"tags":820,"title":823,"type":518},[593],"2024-10-07 16:51:00","BGP-based routing for Chinese and overseas IPs makes transparent proxying more efficient and accurate. Marking overseas destinations with FakeIP lets the main router route traffic more intelligently for smoother connectivity. The sing-box DNS configuration is also refined to handle queries more flexibly and efficiently, improving the overall network experience.","https:\u002F\u002Fblog-img.774352199.xyz\u002FMOmM1s.webp","\u002Ffiddling\u002Fmore-accurate-chnroute",{"text":529,"minutes":817,"time":818,"words":819},3.805,228300,761,[821,822,693,694],"BGP","IP address database","More Accurate Routing for Chinese and Overseas IPs with BGP",{"categories":825,"date":826,"description":827,"image":828,"path":829,"readingTime":830,"recommend":503,"tags":834,"title":839,"type":518},[593],"2025-12-14 14:31:00","A friend’s comment got me interested in simulation games. After buying Microsoft Flight Simulator, I discovered streamed maps and models, another account login, oddly hidden tutorials, and awkward keyboard controls. A flight stick and some tinkering with Pico VR finally rounded out the experience.","https:\u002F\u002Fblog-img.774352199.xyz\u002FlfhEuE.webp","\u002Ffiddling\u002Fmsfs2024-joystick-and-pico",{"text":635,"minutes":831,"time":832,"words":833},4.68,280800,936,[835,836,837,838],"Microsoft Flight Simulator 2024","Thrustmaster TCA","PICO VR","Flight simulation","Adventures with Microsoft Flight Simulator 2024",{"categories":841,"date":842,"description":843,"image":844,"path":845,"readingTime":846,"recommend":503,"tags":850,"title":856,"type":518},[593],"2025-06-05 23:26:00","My frequent phone changes took me from OnePlus to iPhone, and from enjoying tinkering to relying on an ecosystem. At my girlfriend’s suggestion, I recently bought an OPPO Find X8 Ultra to improve my photos. Migrating apps out of Apple’s ecosystem reminded me how uneven Android’s app selection remains and how difficult finding replacements can be. A month of migration has been an exercise in friction and adaptation between platforms.","https:\u002F\u002Fblog-img.774352199.xyz\u002F19NIhZ.webp","\u002Ffiddling\u002Fone-month-using-android",{"text":718,"minutes":847,"time":848,"words":849},7.61,456600,1522,[851,852,853,854,855],"Android","OPPO","iOS","Smartphone impressions","Ecosystem migration","One Month After Switching to Android",{"categories":858,"date":859,"description":860,"image":861,"path":862,"readingTime":863,"recommend":503,"tags":867,"title":873,"type":518},[593],"2026-05-11 22:43:00","I found a trip-planning tool on GitHub Trending and wanted to host my own instance. Since I was getting a new VPS anyway, I might as well install a little extra. Well, quite a lot extra.","https:\u002F\u002Fblog-img.774352199.xyz\u002FBBdDWW.webp","\u002Ffiddling\u002Fone-trek-twenty-stacks",{"text":653,"minutes":864,"time":865,"words":866},5.735,344100,1147,[868,869,870,871,872],"TREK","Docker Compose","Dockge","VPS","Self-hosting","One TREK, Twenty Stacks",{"categories":875,"date":876,"description":877,"image":878,"path":879,"readingTime":880,"recommend":503,"tags":885,"title":889,"type":518},[593],"2025-01-16 23:09:00","OPNsense is an open-source firewall and router with an attractive interface and a comprehensive feature set. After trying several routing setups, I came to appreciate its potential for transparent proxying and traffic routing. Combining it with BGP-based routing offers better security and stability, while its automatically updated IP lists make network management more convenient.","https:\u002F\u002Fblog-img.774352199.xyz\u002FxA8C1E.webp","\u002Ffiddling\u002Fopnsense-transparent-proxy",{"text":881,"minutes":882,"time":883,"words":884},"7 min read",6.47,388200,1294,[886,887,641,888,643,694],"OPNsense","FreeBSD","tun2socks","Transparent Proxying and Traffic Routing with OPNsense",{"categories":891,"date":892,"description":893,"image":894,"path":895,"readingTime":896,"recommend":475,"tags":900,"title":906,"type":518},[593],"2025-03-15 20:35:00","Distinguishing user-defined type names from ordinary variables is a challenge during parsing. A statement such as `a*b;` can be either an arithmetic expression or a declaration. Grammar rules, especially those involving type specifiers, can misidentify variables as types, affecting correctness and readability. The prevalence of declarations without initializers makes this ambiguity especially common.","https:\u002F\u002Fblog-img.774352199.xyz\u002F2VKHK9.webp","\u002Ffiddling\u002Fparser-type-variable-ambiguity",{"text":718,"minutes":897,"time":898,"words":899},7.475,448500,1495,[901,902,903,904,905],"Parsing","GLR","Symbol table","Scope","Disambiguation","Resolving Type Name and Variable Name Ambiguity in Parsing",{"categories":908,"date":909,"description":910,"image":911,"path":912,"readingTime":913,"recommend":503,"tags":917,"title":922,"type":518},[593],"2023-05-24 17:51:09","To install the RISC-V toolchain, first obtain the riscv-gnu-toolchain source. Using `--depth=1` when cloning reduces the download size. Check the README’s Prerequisites section and install the required dependencies. On Debian, a simple package installation command prepares the environment for building the toolchain.","https:\u002F\u002Fblog-img.774352199.xyz\u002FrWNOKx.webp","\u002Ffiddling\u002Fspike-install",{"text":599,"minutes":914,"time":915,"words":916},1.92,115200,384,[626,918,919,920,921],"Spike","riscv-pk","Cross-compilation","Toolchain","Installing the RISC-V Toolchain and Emulator",{"categories":924,"date":925,"description":926,"image":927,"path":928,"readingTime":929,"recommend":933,"tags":934,"title":941,"type":518},[593],"2026-09-09 23:59:00","This counts as NTR too, surely.","https:\u002F\u002Fblog-img.774352199.xyz\u002FNDPUwc.webp","\u002Ffiddling\u002Fsteamdeck-switch-60fps",{"text":718,"minutes":930,"time":931,"words":932},7.345,440700,1469,7,[935,936,937,938,939,940],"Steam Deck","Nintendo Switch","EmuDeck","Eden","Lossless Scaling","Frame generation","A Second Life for the Steam Deck: Switch Emulation and Frame Generation at 60 FPS",{"categories":943,"date":944,"description":945,"image":946,"path":947,"readingTime":948,"recommend":503,"tags":952,"title":957,"type":518},[593],"2024-06-23 15:31:32","The GFW does more than monitor an exit gateway: it inspects international traffic through passive taps, copying inbound and outbound IP packets to a cluster for analysis and filtering. Understanding where and how this happens matters when studying censorship circumvention. Examining the GFW’s network topology helps explain its blocking mechanisms and how to work around them.","https:\u002F\u002Fblog-img.774352199.xyz\u002FsOpJuL.webp","\u002Ffiddling\u002Ftech-about-gfw",{"text":671,"minutes":949,"time":950,"words":951},11.885,713100,2377,[953,954,955,956],"GFW","DNS poisoning","TCP","Internet censorship","How the Great Firewall Works",{"categories":959,"date":960,"description":961,"image":962,"path":963,"readingTime":964,"recommend":503,"tags":968,"title":974,"type":518},[593],"2022-04-16 00:01:28","In Java, using `this` can prevent the compiler from optimizing constants. Although `ab1` and `ab2` in this example appear to refer to the same static final variable `s`, comparing them produces different results. `ab1` concatenates a direct reference to the static variable, while `ab2` accesses it through `this`, preventing the same constant propagation optimization. This illustrates how a small syntactic difference can change compilation behavior.","https:\u002F\u002Fblog-img.774352199.xyz\u002FgKtkYe.webp","\u002Ffiddling\u002Fthis-in-javac-string-concat",{"text":734,"minutes":965,"time":966,"words":967},2.305,138300,461,[969,970,971,972,973],"Java","javac","String concatenation","Constant propagation","Bytecode","How Java’s this Keyword Can Prevent Compile-Time Constant Propagation",{"categories":976,"date":977,"description":978,"image":979,"path":980,"readingTime":981,"recommend":503,"tags":986,"title":992,"type":518},[593],"2025-01-29 21:58:00","A microblog lets you share short thoughts whenever they occur, without the overhead of publishing a full static-blog post. This implementation uses Cloudflare Workers for the backend and KV for storage and management. A Vue component embedded in VitePress displays the updates, adding a lively, interactive element to the blog.","https:\u002F\u002Fblog-img.774352199.xyz\u002FhZX6hr.webp","\u002Ffiddling\u002Fvitepress-memos-component",{"text":982,"minutes":983,"time":984,"words":985},"19 min read",18.96,1137600,3792,[987,988,989,990,991],"VitePress","Vue","Cloudflare Workers","Cloudflare KV","Microblogging","Adding a Microblog to VitePress",{"categories":994,"date":995,"description":996,"image":997,"path":998,"readingTime":999,"recommend":503,"tags":1003,"title":1006,"type":518},[593],"2025-03-15 16:24:00","A Telegram notification on an ordinary afternoon introduced a tempting VPS deal: a direct China Telecom CN2 route and 2.5G bandwidth. The plan includes IPv6, useful for unlocking streaming services, but not every connection needs to go through WARP. My previous script was convenient, yet its effects on speed and traffic routing called for a more flexible solution.","https:\u002F\u002Fblog-img.774352199.xyz\u002FMcjrrF.webp","\u002Ffiddling\u002Fvps-warp-ipv6",{"text":734,"minutes":1000,"time":1001,"words":1002},2.34,140400,468,[1004,1005,871,693,694],"Cloudflare WARP","IPv6","Routing Selected VPS Traffic Through WARP over IPv6",{"categories":1008,"date":1009,"description":1010,"image":1011,"path":1012,"readingTime":1013,"recommend":1017,"tags":1018,"title":1024,"type":518},[593],"2026-08-17 23:59:21","The more I tinker, the more I want to tinker. Still going strong in my old age, still going strong!","https:\u002F\u002Fblog-img.774352199.xyz\u002FqfxB0h.webp","\u002Ffiddling\u002Fxiaomi17-root-and-hide-root",{"text":635,"minutes":1014,"time":1015,"words":1016},4.545,272700,909,6,[1019,1020,1021,1022,1023],"Xiaomi 17","Bootloader","Android Root","Firmware flashing","Root hiding","Making the Xiaomi 17 My Daily Driver: Unlocking, Flashing, Rooting, and Hiding Root",{"categories":1026,"date":1027,"description":1028,"image":1029,"path":1030,"readingTime":1031,"recommend":503,"tags":1035,"title":1040,"type":518},[493],"2022-01-20 22:29:00","Lab 1 asks us to implement a MapReduce system with two core components: a master and workers. This requires a good command of Go RPC and concurrent programming, along with a thorough understanding of the MapReduce workflow. I built two versions, starting with mutex locks and then moving to a more elegant channel-based implementation without explicit locks, whose design is simpler and clearer. The key to understanding the lab is to read the relevant documentation carefully, especially the flowcharts and explanations.","https:\u002F\u002Fblog-img.774352199.xyz\u002FibVwPJ.webp","\u002Fnotes\u002F65840\u002Fmapreducelab",{"text":718,"minutes":1032,"time":1033,"words":1034},7.21,432600,1442,[1036,1037,723,1038,1039],"MIT 6.5840","MapReduce","RPC","Concurrent programming","6.5840 Lab 1: MapReduce",{"categories":1042,"date":1043,"description":1044,"image":1045,"path":1046,"readingTime":1047,"recommend":503,"tags":1051,"title":1055,"type":518},[493],"2022-01-16 17:32:00","MapReduce is an efficient parallel computing model designed to simplify processing large datasets. By defining the two key functions, Map and Reduce, users can break complex tasks into simple operations. The framework automatically handles data distribution and task scheduling, allowing developers to focus on the algorithm rather than low-level details. Its widespread use in distributed systems demonstrates its flexibility and practical value.","https:\u002F\u002Fblog-img.774352199.xyz\u002FApIDdC.webp","\u002Fnotes\u002F65840\u002Fmapreducepaper",{"text":635,"minutes":1048,"time":1049,"words":1050},4.12,247200,824,[1037,1052,517,1053,1054],"Distributed systems","Parallel computing","Fault tolerance","Reading the MapReduce Paper",{"categories":1057,"date":1058,"description":1059,"image":1060,"path":1061,"readingTime":1062,"recommend":503,"tags":1067,"title":1069,"type":518},[493],"2022-12-16 02:06:10","Lab 2A focuses on implementing Raft leader election and heartbeats so that elections and term changes work correctly even under extreme conditions. The lab has four stages and lays the foundation for the distributed key-value store that follows. A design without explicit locks simplifies the Raft struct. The lab instructions provide the necessary background, but compared with the previous lab, this one relies on almost no reference material and places greater emphasis on implementing the system independently.","https:\u002F\u002Fblog-img.774352199.xyz\u002Fc11Uk4.webp","\u002Fnotes\u002F65840\u002Fraftlab2a",{"text":1063,"minutes":1064,"time":1065,"words":1066},"13 min read",12.235,734100,2447,[1036,514,723,1068,515],"Leader election","6.5840 Lab 2A: Leader Election",{"categories":1071,"date":494,"description":495,"image":498,"path":502,"readingTime":1072,"recommend":503,"tags":1073,"title":5,"type":518},[493],{"text":505,"minutes":506,"time":507,"words":508},[514,515,516,517],{"categories":1075,"date":1077,"description":1078,"image":1079,"path":1080,"readingTime":1081,"recommend":1085,"tags":1086,"title":1090,"type":518},[1076],"projects","2021-11-27 14:43:00","MYDB is a personal project exploring and implementing the fundamentals of databases, built in my spare time over a little more than half a month. I picked up some basic knowledge in my university database systems course, though during my internship I mostly used the classes as an excuse to slack off. My candid answers in an interview did not cause too much trouble, but they did make me reconsider what I knew about databases and decide to learn through hands-on practice. That was how this project began.","https:\u002F\u002Fblog-img.774352199.xyz\u002Fxfci2J.webp","\u002Fprojects\u002Fmydb\u002Fmydb0",{"text":635,"minutes":1082,"time":1083,"words":1084},4.15,249000,830,1,[1087,969,1088,1089],"MYDB","Database implementation","Database architecture","MYDB 0. Project Structure and a Few Things I Had to Say",{"categories":1092,"date":1093,"description":1094,"image":1095,"path":1096,"readingTime":1097,"recommend":503,"tags":1101,"title":1104,"type":518},[1076],"2021-11-28 16:10:00","MYDB manages transactions through an XID file. Each transaction has a unique XID, incrementing from 1; XID 0 denotes a super transaction whose state is always committed. TransactionManager maintains this file and records three states: active, committed, and aborted. This mechanism supports accurate transaction state queries and management, providing a foundation for system stability and reliability.","https:\u002F\u002Fblog-img.774352199.xyz\u002FH4zZAK.webp","\u002Fprojects\u002Fmydb\u002Fmydb1",{"text":635,"minutes":1098,"time":1099,"words":1100},4.755,285300,951,[1087,969,1102,1103],"Transaction management","XID","MYDB 1. Starting with the Transaction Manager",{"categories":1106,"date":1107,"description":1108,"image":1109,"path":1110,"readingTime":1111,"recommend":503,"tags":1115,"title":1119,"type":518},[1076],"2021-12-25 18:26:00","MYDB uses a client\u002Fserver architecture similar to MySQL, allowing multiple clients to connect to a server over sockets, execute SQL queries, and receive results. Communication uses a special binary format, though plain text would also be an option for a simpler implementation. The basic transport structure supports effective communication and processing between client and server.","https:\u002F\u002Fblog-img.774352199.xyz\u002FPAHrUZ.webp","\u002Fprojects\u002Fmydb\u002Fmydb10",{"text":635,"minutes":1112,"time":1113,"words":1114},4.305,258300,861,[1087,969,1116,1117,1118],"Socket","Client-server architecture","Communication protocol","MYDB 10. Implementing the Server, Client, and Wire Protocol",{"categories":1121,"date":1122,"description":1123,"image":1124,"path":1125,"readingTime":1126,"recommend":503,"tags":1130,"title":1134,"type":518},[1076],"2021-11-30 23:18:00","The Data Manager (DM) bridges higher-level modules and the filesystem, handling paging and caching while ensuring data safety and recovery. Its cache uses reference counting rather than traditional LRU, aiming for a reusable, efficient foundation for subsequent data operations.","https:\u002F\u002Fblog-img.774352199.xyz\u002FWdIGoG.webp","\u002Fprojects\u002Fmydb\u002Fmydb2",{"text":653,"minutes":1127,"time":1128,"words":1129},5.725,343500,1145,[1087,969,1131,1132,1133],"Reference counting","Cache design","Shared memory","MYDB 2. A Reference-Counted Cache Framework and Shared Byte Arrays",{"categories":1136,"date":1137,"description":1138,"image":1139,"path":1140,"readingTime":1141,"recommend":503,"tags":1145,"title":1148,"type":518},[1076],"2021-12-05 15:28:00","DM abstracts the filesystem into pages and uses them as the unit of reading, writing, and caching. The default page size is 8K, with larger pages available to improve write performance under heavy loads. With the general-purpose cache framework already in place, we now define the page structure and implement efficient page caching.","https:\u002F\u002Fblog-img.774352199.xyz\u002FjlFC4E.webp","\u002Fprojects\u002Fmydb\u002Fmydb3",{"text":635,"minutes":1142,"time":1143,"words":1144},4.7,282000,940,[1087,969,1146,1147],"Data pages","Cache management","MYDB 3. Caching and Managing Data Pages",{"categories":1150,"date":1151,"description":1152,"image":1153,"path":1154,"readingTime":1155,"recommend":503,"tags":1159,"title":1162,"type":518},[1076],"2021-12-08 22:55:00","Log files are essential to MYDB’s design, allowing data to be recovered after a crash. DM logs every operation on underlying data, forming a continuous sequence of records. Stored in a specific binary format with checksums and individual operation records, these logs let the database accurately reconstruct its data on restart and maintain consistency and integrity.","https:\u002F\u002Fblog-img.774352199.xyz\u002FTRcbsj.webp","\u002Fprojects\u002Fmydb\u002Fmydb4",{"text":718,"minutes":1156,"time":1157,"words":1158},7.885,473100,1577,[1087,969,1160,1161],"Database logging","Crash recovery","MYDB 4. Log Files and Recovery Strategies",{"categories":1164,"date":1165,"description":1166,"image":1167,"path":1168,"readingTime":1169,"recommend":503,"tags":1173,"title":1176,"type":518},[1076],"2021-12-11 15:16:00","The page index is an important part of DM, optimizing insertions by caching the free space available on each page. It lets higher-level modules quickly locate a suitable page without a lengthy search, making data operations more efficient. Its implementation works closely with the DataItem abstraction to support efficient database operation.","https:\u002F\u002Fblog-img.774352199.xyz\u002F22PSG1.webp","\u002Fprojects\u002Fmydb\u002Fmydb5",{"text":881,"minutes":1170,"time":1171,"words":1172},6.37,382200,1274,[1087,969,1174,1175],"Page index","Data management","MYDB 5. The Page Index and the Data Manager",{"categories":1178,"date":1179,"description":1180,"image":1181,"path":1182,"readingTime":1183,"recommend":503,"tags":1187,"title":1191,"type":518},[1076],"2021-12-18 14:58:00","VM uses two-phase locking to ensure serializable schedules and introduces multiversion concurrency control (MVCC) to eliminate blocking between reads and writes. This chapter also defines conflicts between database operations, focusing on the interaction between updates and reads as a foundation for understanding transaction isolation levels.","https:\u002F\u002Fblog-img.774352199.xyz\u002F8YzotA.webp","\u002Fprojects\u002Fmydb\u002Fmydb6",{"text":546,"minutes":1184,"time":1185,"words":1186},8.64,518400,1728,[1087,969,1188,1189,1190],"MVCC","Transaction isolation","Two-phase locking","MYDB 6. Record Versions and Transaction Isolation",{"categories":1193,"date":1194,"description":1195,"image":1196,"path":1197,"readingTime":1198,"recommend":503,"tags":1202,"title":1205,"type":518},[1076],"2021-12-23 21:20:00","VM must handle version skipping introduced by MVCC as well as deadlocks. By simply marking a transaction, MYDB can cancel or roll it back and keep data from aborted transactions from affecting others. This design makes concurrent transaction handling more efficient and reliable, avoids the deadlock risks common with traditional 2PL, and improves overall stability and performance.","https:\u002F\u002Fblog-img.774352199.xyz\u002FBF3yDW.webp","\u002Fprojects\u002Fmydb\u002Fmydb7",{"text":718,"minutes":1199,"time":1200,"words":1201},7.265,435900,1453,[1087,969,1203,1204],"Deadlock detection","Version management","MYDB 7. Deadlock Detection and the Version Manager",{"categories":1207,"date":1208,"description":1209,"image":1210,"path":1211,"readingTime":1212,"recommend":503,"tags":1213,"title":1216,"type":518},[1076],"2021-12-24 21:01:00","MYDB implements a clustered index using a B+ tree. IM interacts directly with the Data Manager (DM), bypassing the Version Manager (VM), so index data is written directly to the database file. This chapter details the binary-tree index structure and its basic node fields, including the leaf flag, key count, and sibling identifier, establishing the framework for indexed lookups.","https:\u002F\u002Fblog-img.774352199.xyz\u002Ff92X4o.webp","\u002Fprojects\u002Fmydb\u002Fmydb8",{"text":635,"minutes":1082,"time":1083,"words":1084},[1087,969,1214,1215],"B+ tree","Database indexing","MYDB 8. Index Management",{"categories":1218,"date":1219,"description":1220,"image":1221,"path":1222,"readingTime":1223,"recommend":503,"tags":1227,"title":1230,"type":518},[1076],"2021-12-25 15:44:00","The Table Manager (TBM) manages field and table structures. Parser turns SQL-like statements into structured representations, wrapping their information in the corresponding classes to simplify subsequent operations. This chapter also covers MYDB’s SQL syntax as a foundation for understanding the management process.","https:\u002F\u002Fblog-img.774352199.xyz\u002FzOMyv5.webp","\u002Fprojects\u002Fmydb\u002Fmydb9",{"text":653,"minutes":1224,"time":1225,"words":1226},5.035,302100,1007,[1087,969,1228,1229],"SQL parsing","Table management","MYDB 9. Field and Table Management",1789914053168]