[{"data":1,"prerenderedAt":248},["ShallowReactive",2],{"content:\u002Fprojects\u002Fmydb\u002Fmydb3":3,"surround:\u002Fprojects\u002Fmydb\u002Fmydb3":237},{"id":4,"title":5,"body":6,"categories":214,"date":216,"description":217,"draft":218,"extension":219,"image":220,"meta":221,"navigation":223,"path":224,"permalink":220,"published":220,"readingTime":225,"recommend":220,"references":220,"seo":230,"sitemap":231,"stem":232,"tags":233,"type":235,"__hash__":236},"content\u002Fposts\u002Fprojects\u002Fmydb\u002Fmydb3.md","MYDB 3. 数据页的缓存与管理",{"type":7,"value":8,"toc":204},"minimark",[9,27,31,34,37,40,43,46,57,65,68,74,87,93,99,105,108,114,117,120,124,127,130,133,139,142,148,151,157,164,167,170,173,179,182,188,198],[10,11,12,13,20,21,26],"p",{},"本章涉及代码都在 ",[14,15,19],"a",{"href":16,"rel":17},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Ftree\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm\u002FpageCache",[18],"nofollow","backend\u002Fdm\u002FpageCache"," 和 ",[14,22,25],{"href":23,"rel":24},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Ftree\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm\u002Fpage",[18],"backend\u002Fdm\u002Fpage"," 中。",[28,29,30],"h3",{"id":30},"前言",[10,32,33],{},"本节主要内容就是 DM 模块向下对文件系统的抽象部分。DM 将文件系统抽象成页面，每次对文件系统的读写都是以页面为单位的。同样，从文件系统读进来的数据也是以页面为单位进行缓存的。",[28,35,36],{"id":36},"页面缓存",[10,38,39],{},"这里参考大部分数据库的设计，将默认数据页大小定为 8K。如果想要提升向数据库写入大量数据情况下的性能的话，也可以适当增大这个值。",[10,41,42],{},"上一节我们已经实现了一个通用的缓存框架，那么这一节我们需要缓存页面，就可以直接借用那个缓存的框架了。但是首先，需要定义出页面的结构。注意这个页面是存储在内存中的，与已经持久化到磁盘的抽象页面有区别。",[10,44,45],{},"定义一个页面如下：",[47,48,54],"pre",{"className":49,"code":51,"language":52,"meta":53},[50],"language-java","public class PageImpl implements Page {\n    private int pageNumber;\n    private byte[] data;\n    private boolean dirty;\n    private Lock lock;\n\n    private PageCache pc;\n}\n","java","",[55,56,51],"code",{"__ignoreMap":53},[10,58,59,60,64],{},"其中，pageNumber 是这个页面的页号，",[61,62,63],"strong",{},"该页号从 1 开始","。data 就是这个页实际包含的字节数据。dirty 标志着这个页面是否是脏页面，在缓存驱逐的时候，脏页面需要被写回磁盘。这里保存了一个 PageCache（还未定义）的引用，用来方便在拿到 Page 的引用时可以快速对这个页面的缓存进行释放操作。",[10,66,67],{},"定义页面缓存的接口如下：",[47,69,72],{"className":70,"code":71,"language":52,"meta":53},[50],"public interface PageCache {\n    int newPage(byte[] initData);\n    Page getPage(int pgno) throws Exception;\n    void close();\n    void release(Page page);\n\n    void truncateByBgno(int maxPgno);\n    int getPageNumber();\n    void flushPage(Page pg);\n}\n",[55,73,71],{"__ignoreMap":53},[10,75,76,77,20,80,83,84,86],{},"页面缓存的具体实现类，需要继承抽象缓存框架，并且实现 ",[55,78,79],{"code":79},"getForCache()",[55,81,82],{"code":82},"releaseForCache()"," 两个抽象方法。由于数据源就是文件系统，",[55,85,79],{"code":79}," 直接从文件中读取，并包裹成 Page 即可：",[47,88,91],{"className":89,"code":90,"language":52,"meta":53},[50],"@Override\nprotected Page getForCache(long key) throws Exception {\n    int pgno = (int)key;\n    long offset = PageCacheImpl.pageOffset(pgno);\n\n    ByteBuffer buf = ByteBuffer.allocate(PAGE_SIZE);\n    fileLock.lock();\n    try {\n        fc.position(offset);\n        fc.read(buf);\n    } catch(IOException e) {\n        Panic.panic(e);\n    }\n    fileLock.unlock();\n    return new PageImpl(pgno, buf.array(), this);\n}\n\nprivate static long pageOffset(int pgno) {\n    \u002F\u002F 页号从 1 开始\n    return (pgno-1) * PAGE_SIZE;\n}\n",[55,92,90],{"__ignoreMap":53},[10,94,95,96,98],{},"而 ",[55,97,82],{"code":82}," 驱逐页面时，也只需要根据页面是否是脏页面，来决定是否需要写回文件系统：",[47,100,103],{"className":101,"code":102,"language":52,"meta":53},[50],"@Override\nprotected void releaseForCache(Page pg) {\n    if(pg.isDirty()) {\n        flush(pg);\n        pg.setDirty(false);\n    }\n}\n\nprivate void flush(Page pg) {\n    int pgno = pg.getPageNumber();\n    long offset = pageOffset(pgno);\n\n    fileLock.lock();\n    try {\n        ByteBuffer buf = ByteBuffer.wrap(pg.getData());\n        fc.position(offset);\n        fc.write(buf);\n        fc.force(false);\n    } catch(IOException e) {\n        Panic.panic(e);\n    } finally {\n        fileLock.unlock();\n    }\n}\n",[55,104,102],{"__ignoreMap":53},[10,106,107],{},"PageCache 还使用了一个 AtomicInteger，来记录了当前打开的数据库文件有多少页。这个数字在数据库文件被打开时就会被计算，并在新建页面时自增。",[47,109,112],{"className":110,"code":111,"language":52,"meta":53},[50],"public int newPage(byte[] initData) {\n    int pgno = pageNumbers.incrementAndGet();\n    Page pg = new PageImpl(pgno, initData, null);\n    flush(pg);  \u002F\u002F 新建的页面需要立刻写回\n    return pgno;\n}\n",[55,113,111],{"__ignoreMap":53},[10,115,116],{},"提一点，同一条数据是不允许跨页存储的，这一点会从后面的章节中体现。这意味着，单条数据的大小不能超过数据库页面的大小。",[28,118,119],{"id":119},"数据页管理",[121,122,123],"h4",{"id":123},"第一页",[10,125,126],{},"数据库文件的第一页，通常用作一些特殊用途，比如存储一些元数据，用来启动检查什么的。MYDB 的第一页，只是用来做启动检查。具体的原理是，在每次数据库启动时，会生成一串随机字节，存储在 100 ~ 107 字节。在数据库正常关闭时，会将这串字节，拷贝到第一页的 108 ~ 115 字节。",[10,128,129],{},"这样数据库在每次启动时，就会检查第一页两处的字节是否相同，以此来判断上一次是否正常关闭。如果是异常关闭，就需要执行数据的恢复流程。",[10,131,132],{},"启动时设置初始字节：",[47,134,137],{"className":135,"code":136,"language":52,"meta":53},[50],"public static void setVcOpen(Page pg) {\n    pg.setDirty(true);\n    setVcOpen(pg.getData());\n}\n\nprivate static void setVcOpen(byte[] raw) {\n    System.arraycopy(RandomUtil.randomBytes(LEN_VC), 0, raw, OF_VC, LEN_VC);\n}\n",[55,138,136],{"__ignoreMap":53},[10,140,141],{},"关闭时拷贝字节：",[47,143,146],{"className":144,"code":145,"language":52,"meta":53},[50],"public static void setVcClose(Page pg) {\n    pg.setDirty(true);\n    setVcClose(pg.getData());\n}\n\nprivate static void setVcClose(byte[] raw) {\n    System.arraycopy(raw, OF_VC, raw, OF_VC+LEN_VC, LEN_VC);\n}\n",[55,147,145],{"__ignoreMap":53},[10,149,150],{},"校验字节：",[47,152,155],{"className":153,"code":154,"language":52,"meta":53},[50],"public static boolean checkVc(Page pg) {\n    return checkVc(pg.getData());\n}\n\nprivate static boolean checkVc(byte[] raw) {\n    return Arrays.equals(Arrays.copyOfRange(raw, OF_VC, OF_VC+LEN_VC), Arrays.copyOfRange(raw, OF_VC+LEN_VC, OF_VC+2*LEN_VC));\n}\n",[55,156,154],{"__ignoreMap":53},[10,158,159,160,163],{},"似乎就是这个 ",[55,161,162],{"code":162},"Arrays.compare()"," 方法不兼容 JDK8，可以使用其他等价的方法替换。",[121,165,166],{"id":166},"普通页",[10,168,169],{},"MYDB 对于普通数据页的管理就比较简单了。一个普通页面以一个 2 字节无符号数起始，表示这一页的空闲位置的偏移。剩下的部分都是实际存储的数据。",[10,171,172],{},"所以对普通页的管理，基本都是围绕着对 FSO（Free Space Offset）进行的。例如向页面插入数据：",[47,174,177],{"className":175,"code":176,"language":52,"meta":53},[50],"\u002F\u002F 将 raw 插入 pg 中，返回插入位置\npublic static short insert(Page pg, byte[] raw) {\n    pg.setDirty(true);\n    short offset = getFSO(pg.getData());\n    System.arraycopy(raw, 0, pg.getData(), offset, raw.length);\n    setFSO(pg.getData(), (short)(offset + raw.length));\n    return offset;\n}\n",[55,178,176],{"__ignoreMap":53},[10,180,181],{},"在写入之前获取 FSO，来确定写入的位置，并在写入之后更新 FSO。FSO 的操作如下：",[47,183,186],{"className":184,"code":185,"language":52,"meta":53},[50],"private static void setFSO(byte[] raw, short ofData) {\n    System.arraycopy(Parser.short2Byte(ofData), 0, raw, OF_FREE, OF_DATA);\n}\n\n\u002F\u002F 获取 pg 的 FSO\npublic static short getFSO(Page pg) {\n    return getFSO(pg.getData());\n}\n\nprivate static short getFSO(byte[] raw) {\n    return Parser.parseShort(Arrays.copyOfRange(raw, 0, 2));\n}\n\n\u002F\u002F 获取页面的空闲空间大小\npublic static int getFreeSpace(Page pg) {\n    return PageCache.PAGE_SIZE - (int)getFSO(pg.getData());\n}\n",[55,187,185],{"__ignoreMap":53},[10,189,190,191,20,194,197],{},"剩余两个函数 ",[55,192,193],{"code":193},"recoverInsert()",[55,195,196],{"code":196},"recoverUpdate()"," 用于在数据库崩溃后重新打开时，恢复例程直接插入数据以及修改数据使用。",[47,199,202],{"className":200,"code":201,"language":52,"meta":53},[50],"\u002F\u002F 将 raw 插入 pg 中的 offset 位置，并将 pg 的 offset 设置为较大的 offset\npublic static void recoverInsert(Page pg, byte[] raw, short offset) {\n    pg.setDirty(true);\n    System.arraycopy(raw, 0, pg.getData(), offset, raw.length);\n\n    short rawFSO = getFSO(pg.getData());\n    if(rawFSO \u003C offset + raw.length) {\n        setFSO(pg.getData(), (short)(offset+raw.length));\n    }\n}\n\n\u002F\u002F 将 raw 插入 pg 中的 offset 位置，不更新 update\npublic static void recoverUpdate(Page pg, byte[] raw, short offset) {\n    pg.setDirty(true);\n    System.arraycopy(raw, 0, pg.getData(), offset, raw.length);\n}\n",[55,203,201],{"__ignoreMap":53},{"title":53,"searchDepth":205,"depth":205,"links":206},4,[207,209,210],{"id":30,"depth":208,"text":30},3,{"id":36,"depth":208,"text":36},{"id":119,"depth":208,"text":119,"children":211},[212,213],{"id":123,"depth":205,"text":123},{"id":166,"depth":205,"text":166},[215],"项目","2021-12-05 15:28:00","DM 模块将文件系统抽象为页面，并以此为单位进行数据的读写和缓存。默认设置的数据页大小为8K，便于在高负载情况下提升写入性能。借助已实现的通用缓存框架，接下来将专注于具体定义页面结构，以实现高效的页面缓存管理。",false,"md",null,{"slots":222},{},true,"\u002Fprojects\u002Fmydb\u002Fmydb3",{"text":226,"minutes":227,"time":228,"words":229},"7 min read",6.96,417600,1392,{"title":5,"description":217},{"loc":224},"posts\u002Fprojects\u002Fmydb\u002Fmydb3",[52,234],"mydb","tech","371OtDmMz5ALuRJe8LvuzicSfaI44DIpFLKiwV0pc8Q",[238,243],{"title":239,"path":240,"stem":241,"date":242,"type":235,"children":-1},"MYDB 2. 引用计数缓存框架和共享内存数组","\u002Fprojects\u002Fmydb\u002Fmydb2","posts\u002Fprojects\u002Fmydb\u002Fmydb2","2021-11-30 23:18:00",{"title":244,"path":245,"stem":246,"date":247,"type":235,"children":-1},"MYDB 4. 日志文件与恢复策略","\u002Fprojects\u002Fmydb\u002Fmydb4","posts\u002Fprojects\u002Fmydb\u002Fmydb4","2021-12-08 22:55:00",1787554445891]