[{"data":1,"prerenderedAt":367},["ShallowReactive",2],{"content:\u002Fprojects\u002Fmydb\u002Fmydb4":3,"surround:\u002Fprojects\u002Fmydb\u002Fmydb4":356},{"id":4,"title":5,"body":6,"categories":333,"date":335,"description":336,"draft":337,"extension":338,"image":339,"meta":340,"navigation":342,"path":343,"permalink":339,"published":339,"readingTime":344,"recommend":339,"references":339,"seo":349,"sitemap":350,"stem":351,"tags":352,"type":354,"__hash__":355},"content\u002Fposts\u002Fprojects\u002Fmydb\u002Fmydb4.md","MYDB 4. 日志文件与恢复策略",{"type":7,"value":8,"toc":322},"minimark",[9,27,31,34,37,40,51,54,57,63,66,69,77,80,94,100,103,109,112,118,121,124,127,130,136,139,142,152,155,161,165,168,177,180,191,194,205,208,211,214,217,223,226,231,234,240,243,248,251,259,262,267,270,278,281,284,287,293,296,302,305,311],[10,11,12,13,20,21,26],"p",{},"本章涉及代码都在 ",[14,15,19],"a",{"href":16,"rel":17},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Ftree\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm\u002Flogger",[18],"nofollow","backend\u002Fdm\u002Flogger"," 和 ",[14,22,25],{"href":23,"rel":24},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Fblob\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm\u002FRecover.java",[18],"backend\u002Fdm\u002FRecover.java"," 中。",[28,29,30],"h3",{"id":30},"前言",[10,32,33],{},"MYDB 提供了崩溃后的数据恢复功能。DM 层在每次对底层数据操作时，都会记录一条日志到磁盘上。在数据库奔溃之后，再次启动时，可以根据日志的内容，恢复数据文件，保证其一致性。",[28,35,36],{"id":36},"日志读写",[10,38,39],{},"日志的二进制文件，按照如下的格式进行排布：",[41,42,47],"pre",{"className":43,"code":45,"language":46},[44],"language-text","[XChecksum][Log1][Log2][Log3]...[LogN][BadTail]\n","text",[48,49,45],"code",{"__ignoreMap":50},"",[10,52,53],{},"其中 XChecksum 是一个四字节的整数，是对后续所有日志计算的校验和。Log1 ~ LogN 是常规的日志数据，BadTail 是在数据库崩溃时，没有来得及写完的日志数据，这个 BadTail 不一定存在。",[10,55,56],{},"每条日志的格式如下：",[41,58,61],{"className":59,"code":60,"language":46},[44],"[Size][Checksum][Data]\n",[48,62,60],{"__ignoreMap":50},[10,64,65],{},"其中，Size 是一个四字节整数，标识了 Data 段的字节数。Checksum 则是该条日志的校验和。",[10,67,68],{},"单条日志的校验和，其实就是通过一个指定的种子实现的：",[41,70,75],{"className":71,"code":73,"language":74,"meta":50},[72],"language-java","private int calChecksum(int xCheck, byte[] log) {\n    for (byte b : log) {\n        xCheck = xCheck * SEED + b;\n    }\n    return xCheck;\n}\n","java",[48,76,73],{"__ignoreMap":50},[10,78,79],{},"这样，对所有日志求出校验和，求和就能得到日志文件的校验和了。",[10,81,82,83,86,87,89,90,93],{},"Logger 被实现成迭代器模式，通过 ",[48,84,85],{"code":85},"next()"," 方法，不断地从文件中读取下一条日志，并将其中的 Data 解析出来并返回。",[48,88,85],{"code":85}," 方法的实现主要依靠 ",[48,91,92],{"code":92},"internNext()","，大致如下，其中 position 是当前日志文件读到的位置偏移：",[41,95,98],{"className":96,"code":97,"language":74,"meta":50},[72],"private byte[] internNext() {\n    if(position + OF_DATA >= fileSize) {\n        return null;\n    }\n    \u002F\u002F 读取 size\n    ByteBuffer tmp = ByteBuffer.allocate(4);\n    fc.position(position);\n    fc.read(tmp);\n    int size = Parser.parseInt(tmp.array());\n    if(position + size + OF_DATA > fileSize) {\n        return null;\n    }\n\n    \u002F\u002F 读取 checksum+data\n    ByteBuffer buf = ByteBuffer.allocate(OF_DATA + size);\n    fc.position(position);\n    fc.read(buf);\n    byte[] log = buf.array();\n\n    \u002F\u002F 校验 checksum\n    int checkSum1 = calChecksum(0, Arrays.copyOfRange(log, OF_DATA, log.length));\n    int checkSum2 = Parser.parseInt(Arrays.copyOfRange(log, OF_CHECKSUM, OF_DATA));\n    if(checkSum1 != checkSum2) {\n        return null;\n    }\n    position += log.length;\n    return log;\n}\n",[48,99,97],{"__ignoreMap":50},[10,101,102],{},"在打开一个日志文件时，需要首先校验日志文件的 XChecksum，并移除文件尾部可能存在的 BadTail，由于 BadTail 该条日志尚未写入完成，文件的校验和也就不会包含该日志的校验和，去掉 BadTail 即可保证日志文件的一致性。",[41,104,107],{"className":105,"code":106,"language":74,"meta":50},[72],"private void checkAndRemoveTail() {\n    rewind();\n\n    int xCheck = 0;\n    while(true) {\n        byte[] log = internNext();\n        if(log == null) break;\n        xCheck = calChecksum(xCheck, log);\n    }\n    if(xCheck != xChecksum) {\n        Panic.panic(Error.BadLogFileException);\n    }\n\n    \u002F\u002F 截断文件到正常日志的末尾\n    truncate(position);\n    rewind();\n}\n",[48,108,106],{"__ignoreMap":50},[10,110,111],{},"向日志文件写入日志时，也是首先将数据包裹成日志格式，写入文件后，再更新文件的校验和，更新校验和时，会刷新缓冲区，保证内容写入磁盘。",[41,113,116],{"className":114,"code":115,"language":74,"meta":50},[72],"public void log(byte[] data) {\n    byte[] log = wrapLog(data);\n    ByteBuffer buf = ByteBuffer.wrap(log);\n    lock.lock();\n    try {\n        fc.position(fc.size());\n        fc.write(buf);\n    } catch(IOException e) {\n        Panic.panic(e);\n    } finally {\n        lock.unlock();\n    }\n    updateXChecksum(log);\n}\n\nprivate void updateXChecksum(byte[] log) {\n    this.xChecksum = calChecksum(this.xChecksum, log);\n    fc.position(0);\n    fc.write(ByteBuffer.wrap(Parser.int2Byte(xChecksum)));\n    fc.force(false);\n}\n\nprivate byte[] wrapLog(byte[] data) {\n    byte[] checksum = Parser.int2Byte(calChecksum(0, data));\n    byte[] size = Parser.int2Byte(data.length);\n    return Bytes.concat(size, checksum, data);\n}\n",[48,117,115],{"__ignoreMap":50},[28,119,120],{"id":120},"恢复策略",[10,122,123],{},"恢复策略来自于 NYADB2 的恢复策略，比较烧脑（我感觉）。",[10,125,126],{},"DM 为上层模块，提供了两种操作，分别是插入新数据（I）和更新现有数据（U）。至于为啥没有删除数据，这个会在 VM 一节叙述。",[10,128,129],{},"DM 的日志策略很简单，一句话就是：",[131,132,133],"blockquote",{},[10,134,135],{},"在进行 I 和 U 操作之前，必须先进行对应的日志操作，在保证日志写入磁盘后，才进行数据操作。",[10,137,138],{},"这个日志策略，使得 DM 对于数据操作的磁盘同步，可以更加随意。日志在数据操作之前，保证到达了磁盘，那么即使该数据操作最后没有来得及同步到磁盘，数据库就发生了崩溃，后续也可以通过磁盘上的日志恢复该数据。",[10,140,141],{},"对于两种数据操作，DM 记录的日志如下：",[143,144,145,149],"ul",{},[146,147,148],"li",{},"(Ti, I, A, x)，表示事务 Ti 在 A 位置插入了一条数据 x",[146,150,151],{},"(Ti, U, A, oldx, newx)，表示事务 Ti 将 A 位置的数据，从 oldx 更新成 newx",[10,153,154],{},"我们首先不考虑并发的情况，那么在某一时刻，只可能有一个事务在操作数据库。日志会看起来像下面那样：",[41,156,159],{"className":157,"code":158,"language":46},[44],"(Ti, x, x), ..., (Ti, x, x), (Tj, x, x), ..., (Tj, x, x), (Tk, x, x), ..., (Tk, x, x)\n",[48,160,158],{"__ignoreMap":50},[162,163,164],"h4",{"id":164},"单线程",[10,166,167],{},"由于单线程，Ti、Tj 和 Tk 的日志永远不会相交。这种情况下利用日志恢复很简单，假设日志中最后一个事务是 Ti：",[169,170,171,174],"ol",{},[146,172,173],{},"对 Ti 之前所有的事务的日志，进行重做（redo）",[146,175,176],{},"接着检查 Ti 的状态（XID 文件），如果 Ti 的状态是已完成（包括 committed 和 aborted），就将 Ti 重做，否则进行撤销（undo）",[10,178,179],{},"接着，是如何对事务 T 进行 redo：",[169,181,182,185,188],{},[146,183,184],{},"正序扫描事务 T 的所有日志",[146,186,187],{},"如果日志是插入操作 (Ti, I, A, x)，就将 x 重新插入 A 位置",[146,189,190],{},"如果日志是更新操作 (Ti, U, A, oldx, newx)，就将 A 位置的值设置为 newx",[10,192,193],{},"undo 也很好理解：",[169,195,196,199,202],{},[146,197,198],{},"倒序扫描事务 T 的所有日志",[146,200,201],{},"如果日志是插入操作 (Ti, I, A, x)，就将 A 位置的数据删除",[146,203,204],{},"如果日志是更新操作 (Ti, U, A, oldx, newx)，就将 A 位置的值设置为 oldx",[10,206,207],{},"注意，MYDB 中其实没有真正的删除操作，对于插入操作的 undo，只是将其中的标志位设置为 invalid。对于删除的探讨将在 VM 一节中进行。",[162,209,210],{"id":210},"多线程",[10,212,213],{},"经过以上的操作，就能保证了 MYDB 在单线程下的恢复性。对于多线程的情况下呢？我们来考虑下面的两种情况。",[10,215,216],{},"第一种：",[41,218,221],{"className":219,"code":220,"language":46},[44],"T1 begin\nT2 begin\nT2 U(x)\nT1 R(x)\n...\nT1 commit\nMYDB break down\n",[48,222,220],{"__ignoreMap":50},[10,224,225],{},"在系统崩溃时，T2 仍然是活跃状态。那么当数据库重新启动，执行恢复例程时，会撤销 T2，它对数据库的影响会被消除。但是由于 T1 读取了 T2 更新的值，既然 T2 被撤销，那么 T1 也应当被撤销。这种情况，就是级联回滚。但是，T1 已经 commit 了，所有 commit 的事务的影响，应当被持久化。这里就造成了矛盾。所以这里需要保证：",[131,227,228],{},[10,229,230],{},"规定 1：正在进行的事务，不会读取其他任何未提交的事务产生的数据。",[10,232,233],{},"第二种情况，假设 x 的初值是 0",[41,235,238],{"className":236,"code":237,"language":46},[44],"T1 begin\nT2 begin\nT1 set x = x+1 \u002F\u002F 产生的日志为 (T1, U, A, 0, 1)\nT2 set x = x+1 \u002F\u002F 产生的日志为 (T1, U, A, 1, 2)\nT2 commit\nMYDB break down\n",[48,239,237],{"__ignoreMap":50},[10,241,242],{},"在系统崩溃时，T1 仍然是活跃状态。那么当数据库重新启动，执行恢复例程时，会对 T1 进行撤销，对 T2 进行重做，但是，无论撤销和重做的先后顺序如何，x 最后的结果，要么是 0，要么是 2，这都是错误的。",[131,244,245],{},[10,246,247],{},"出现这种问题的原因，归根结底是因为我们的日志太过简单，仅仅记录了\"前相\"和\"后相\". 并单纯的依靠\"前相\"undo, 依靠\"后相\"redo. 这种简单的日志方式和恢复方式，并不能涵盖住所有数据库操作形成的语义",[10,249,250],{},"解决方法有两种：",[169,252,253,256],{},[146,254,255],{},"增加日志种类",[146,257,258],{},"限制数据库操作",[10,260,261],{},"MYDB 采用的是限制数据库操作，需要保证：",[131,263,264],{},[10,265,266],{},"规定 2：正在进行的事务，不会修改其他任何未提交的事务修改或产生的数据。",[10,268,269],{},"在 MYDB 中，由于 VM 的存在，传递到 DM 层，真正执行的操作序列，都可以保证规定 1 和规定 2。VM 如何保证这两条规定，会在 VM 层一节中说明（VM 的坑还挺大）。有了这两条规定，并发情况下日志的恢复也就很简单了：",[169,271,272,275],{},[146,273,274],{},"重做所有崩溃时已完成（committed 或 aborted）的事务",[146,276,277],{},"撤销所有崩溃时未完成（active）的事务",[10,279,280],{},"在恢复后，数据库就会恢复到所有已完成事务结束，所有未完成事务尚未开始的状态。",[162,282,283],{"id":283},"实现",[10,285,286],{},"首先规定两种日志的格式：",[41,288,291],{"className":289,"code":290,"language":74,"meta":50},[72],"private static final byte LOG_TYPE_INSERT = 0;\nprivate static final byte LOG_TYPE_UPDATE = 1;\n\n\u002F\u002F updateLog:\n\u002F\u002F [LogType] [XID] [UID] [OldRaw] [NewRaw]\n\n\u002F\u002F insertLog:\n\u002F\u002F [LogType] [XID] [Pgno] [Offset] [Raw]\n",[48,292,290],{"__ignoreMap":50},[10,294,295],{},"和原理中描述的类似，recover 例程主要也是两步：重做所有已完成事务，撤销所有未完成事务：",[41,297,300],{"className":298,"code":299,"language":74,"meta":50},[72],"private static void redoTranscations(TransactionManager tm, Logger lg, PageCache pc) {\n    lg.rewind();\n    while(true) {\n        byte[] log = lg.next();\n        if(log == null) break;\n        if(isInsertLog(log)) {\n            InsertLogInfo li = parseInsertLog(log);\n            long xid = li.xid;\n            if(!tm.isActive(xid)) {\n                doInsertLog(pc, log, REDO);\n            }\n        } else {\n            UpdateLogInfo xi = parseUpdateLog(log);\n            long xid = xi.xid;\n            if(!tm.isActive(xid)) {\n                doUpdateLog(pc, log, REDO);\n            }\n        }\n    }\n}\n\nprivate static void undoTranscations(TransactionManager tm, Logger lg, PageCache pc) {\n    Map\u003CLong, List\u003Cbyte[]>> logCache = new HashMap\u003C>();\n    lg.rewind();\n    while(true) {\n        byte[] log = lg.next();\n        if(log == null) break;\n        if(isInsertLog(log)) {\n            InsertLogInfo li = parseInsertLog(log);\n            long xid = li.xid;\n            if(tm.isActive(xid)) {\n                if(!logCache.containsKey(xid)) {\n                    logCache.put(xid, new ArrayList\u003C>());\n                }\n                logCache.get(xid).add(log);\n            }\n        } else {\n            UpdateLogInfo xi = parseUpdateLog(log);\n            long xid = xi.xid;\n            if(tm.isActive(xid)) {\n                if(!logCache.containsKey(xid)) {\n                    logCache.put(xid, new ArrayList\u003C>());\n                }\n                logCache.get(xid).add(log);\n            }\n        }\n    }\n\n    \u002F\u002F 对所有 active log 进行倒序 undo\n    for(Entry\u003CLong, List\u003Cbyte[]>> entry : logCache.entrySet()) {\n        List\u003Cbyte[]> logs = entry.getValue();\n        for (int i = logs.size()-1; i >= 0; i --) {\n            byte[] log = logs.get(i);\n            if(isInsertLog(log)) {\n                doInsertLog(pc, log, UNDO);\n            } else {\n                doUpdateLog(pc, log, UNDO);\n            }\n        }\n        tm.abort(entry.getKey());\n    }\n}\n",[48,301,299],{"__ignoreMap":50},[10,303,304],{},"updateLog 和 insertLog 的重做和撤销处理，分别合并成一个方法来实现：",[41,306,309],{"className":307,"code":308,"language":74,"meta":50},[72],"private static void doUpdateLog(PageCache pc, byte[] log, int flag) {\n    int pgno;\n    short offset;\n    byte[] raw;\n    if(flag == REDO) {\n        UpdateLogInfo xi = parseUpdateLog(log);\n        pgno = xi.pgno;\n        offset = xi.offset;\n        raw = xi.newRaw;\n    } else {\n        UpdateLogInfo xi = parseUpdateLog(log);\n        pgno = xi.pgno;\n        offset = xi.offset;\n        raw = xi.oldRaw;\n    }\n    Page pg = null;\n    try {\n        pg = pc.getPage(pgno);\n    } catch (Exception e) {\n        Panic.panic(e);\n    }\n    try {\n        PageX.recoverUpdate(pg, raw, offset);\n    } finally {\n        pg.release();\n    }\n}\n\nprivate static void doInsertLog(PageCache pc, byte[] log, int flag) {\n    InsertLogInfo li = parseInsertLog(log);\n    Page pg = null;\n    try {\n        pg = pc.getPage(li.pgno);\n    } catch(Exception e) {\n        Panic.panic(e);\n    }\n    try {\n        if(flag == UNDO) {\n            DataItem.setDataItemRawInvalid(li.raw);\n        }\n        PageX.recoverInsert(pg, li.raw, li.offset);\n    } finally {\n        pg.release();\n    }\n}\n",[48,310,308],{"__ignoreMap":50},[10,312,313,314,317,318,321],{},"注意，",[48,315,316],{"code":316},"doInsertLog()"," 方法中的删除，使用的是 ",[48,319,320],{"code":320},"DataItem.setDataItemRawInvalid(li.raw);","，dataItem 将在下一节中说明，大致的作用，就是将该条 DataItem 的有效位设置为无效，来进行逻辑删除。",{"title":50,"searchDepth":323,"depth":323,"links":324},4,[325,327,328],{"id":30,"depth":326,"text":30},3,{"id":36,"depth":326,"text":36},{"id":120,"depth":326,"text":120,"children":329},[330,331,332],{"id":164,"depth":323,"text":164},{"id":210,"depth":323,"text":210},{"id":283,"depth":323,"text":283},[334],"项目","2021-12-08 22:55:00","在 MYDB 的设计中，日志文件扮演着至关重要的角色，确保数据在崩溃后能够顺利恢复。每次操作底层数据时，DM 层都会生成并记录日志，形成一条连续的日志链。这些日志以特定的二进制格式存储，包含校验和和各个操作记录，确保在系统重启时能够精准地重建数据状态，维护数据的一致性与完整性。",false,"md",null,{"slots":341},{},true,"\u002Fprojects\u002Fmydb\u002Fmydb4",{"text":345,"minutes":346,"time":347,"words":348},"13 min read",12.12,727200,2424,{"title":5,"description":336},{"loc":343},"posts\u002Fprojects\u002Fmydb\u002Fmydb4",[74,353],"mydb","tech","gUiPG5KVrI7PpPzG4RuLyzs9S1zrDH2t7RqJHC0qu_I",[357,362],{"title":358,"path":359,"stem":360,"date":361,"type":354,"children":-1},"MYDB 3. 数据页的缓存与管理","\u002Fprojects\u002Fmydb\u002Fmydb3","posts\u002Fprojects\u002Fmydb\u002Fmydb3","2021-12-05 15:28:00",{"title":363,"path":364,"stem":365,"date":366,"type":354,"children":-1},"MYDB 5. 页面索引与 DM 的实现","\u002Fprojects\u002Fmydb\u002Fmydb5","posts\u002Fprojects\u002Fmydb\u002Fmydb5","2021-12-11 15:16:00",1787554445837]