[{"data":1,"prerenderedAt":322},["ShallowReactive",2],{"content:\u002Fprojects\u002Fmydb\u002Fmydb5":3,"surround:\u002Fprojects\u002Fmydb\u002Fmydb5":311},{"id":4,"title":5,"body":6,"categories":288,"date":290,"description":291,"draft":292,"extension":293,"image":294,"meta":295,"navigation":297,"path":298,"permalink":294,"published":294,"readingTime":299,"recommend":294,"references":294,"seo":304,"sitemap":305,"stem":306,"tags":307,"type":309,"__hash__":310},"content\u002Fposts\u002Fprojects\u002Fmydb\u002Fmydb5.md","MYDB 5. 页面索引与 DM 的实现",{"type":7,"value":8,"toc":280},"minimark",[9,33,37,40,43,46,49,52,63,66,72,75,78,84,87,93,96,100,103,106,112,115,118,126,129,136,142,157,163,168,171,177,181,184,191,197,200,206,209,215,218,224,234,239,245,250,256,259,265,268,271,277],[10,11,12,13,20,21,26,27,32],"p",{},"本章涉及代码都在 ",[14,15,19],"a",{"href":16,"rel":17},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Ftree\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm\u002FpageIndex",[18],"nofollow","backend\u002Fdm\u002FpageIndex","、",[14,22,25],{"href":23,"rel":24},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Ftree\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm\u002FdataItem",[18],"backend\u002Fdm\u002FdataItem"," 和 ",[14,28,31],{"href":29,"rel":30},"https:\u002F\u002Fgithub.com\u002FCN-GuoZiyang\u002FMYDB\u002Ftree\u002Fmaster\u002Fsrc\u002Fmain\u002Fjava\u002Ftop\u002Fguoziyang\u002Fmydb\u002Fbackend\u002Fdm",[18],"backend\u002Fdm"," 中。",[34,35,36],"h3",{"id":36},"前言",[10,38,39],{},"本节将为 DM 层做收尾，介绍一个实现简单的页面索引。并且实现了 DM 层对于上层的抽象：DataItem。",[34,41,42],{"id":42},"页面索引",[10,44,45],{},"页面索引，缓存了每一页的空闲空间。用于在上层模块进行插入操作时，能够快速找到一个合适空间的页面，而无需从磁盘或者缓存中检查每一个页面的信息。",[10,47,48],{},"MYDB 用一个比较粗略的算法实现了页面索引，将一页的空间划分成了 40 个区间。在启动时，就会遍历所有的页面信息，获取页面的空闲空间，安排到这 40 个区间中。insert 在请求一个页时，会首先将所需的空间向上取整，映射到某一个区间，随后取出这个区间的任何一页，都可以满足需求。",[10,50,51],{},"PageIndex 的实现也很简单，一个 List 类型的数组。",[53,54,60],"pre",{"className":55,"code":57,"language":58,"meta":59},[56],"language-java","public class PageIndex {\n    \u002F\u002F 将一页划成 40 个区间\n    private static final int INTERVALS_NO = 40;\n    private static final int THRESHOLD = PageCache.PAGE_SIZE \u002F INTERVALS_NO;\n\n    private List[] lists;\n}\n","java","",[61,62,57],"code",{"__ignoreMap":59},[10,64,65],{},"从 PageIndex 中获取页面也很简单，算出区间号，直接取即可：",[53,67,70],{"className":68,"code":69,"language":58,"meta":59},[56],"public PageInfo select(int spaceSize) {\n    int number = spaceSize \u002F THRESHOLD;\n    if(number \u003C INTERVALS_NO) number ++;\n    while(number \u003C= INTERVALS_NO) {\n        if(lists[number].size() == 0) {\n            number ++;\n            continue;\n        }\n        return lists[number].remove(0);\n    }\n    return null;\n}\n",[61,71,69],{"__ignoreMap":59},[10,73,74],{},"返回的 PageInfo 中包含页号和空闲空间大小的信息。",[10,76,77],{},"可以注意到，被选择的页，会直接从 PageIndex 中移除，这意味着，同一个页面是不允许并发写的。在上层模块使用完这个页面后，需要将其重新插入 PageIndex：",[53,79,82],{"className":80,"code":81,"language":58,"meta":59},[56],"public void add(int pgno, int freeSpace) {\n    int number = freeSpace \u002F THRESHOLD;\n    lists[number].add(new PageInfo(pgno, freeSpace));\n}\n",[61,83,81],{"__ignoreMap":59},[10,85,86],{},"在 DataManager 被创建时，需要获取所有页面并填充 PageIndex：",[53,88,91],{"className":89,"code":90,"language":58,"meta":59},[56],"\u002F\u002F 初始化 pageIndex\nvoid fillPageIndex() {\n    int pageNumber = pc.getPageNumber();\n    for(int i = 2; i \u003C= pageNumber; i ++) {\n        Page pg = null;\n        try {\n            pg = pc.getPage(i);\n        } catch (Exception e) {\n            Panic.panic(e);\n        }\n        pIndex.add(pg.getPageNumber(), PageX.getFreeSpace(pg));\n        pg.release();\n    }\n}\n",[61,92,90],{"__ignoreMap":59},[10,94,95],{},"注意在使用完 Page 后需要及时 release，否则可能会撑爆缓存。",[34,97,99],{"id":98},"dataitem","DataItem",[10,101,102],{},"DataItem 是 DM 层向上层提供的数据抽象。上层模块通过地址，向 DM 请求到对应的 DataItem，再获取到其中的数据。",[10,104,105],{},"DataItem 的实现很简单：",[53,107,110],{"className":108,"code":109,"language":58,"meta":59},[56],"public class DataItemImpl implements DataItem {\n    private SubArray raw;\n    private byte[] oldRaw;\n    private DataManagerImpl dm;\n    private long uid;\n    private Page pg;\n}\n",[61,111,109],{"__ignoreMap":59},[10,113,114],{},"保存一个 dm 的引用是因为其释放依赖 dm 的释放（dm 同时实现了缓存接口，用于缓存 DataItem），以及修改数据时落日志。",[10,116,117],{},"DataItem 中保存的数据，结构如下：",[53,119,124],{"className":120,"code":122,"language":123},[121],"language-text","[ValidFlag] [DataSize] [Data]\n","text",[61,125,122],{"__ignoreMap":59},[10,127,128],{},"其中 ValidFlag 占用 1 字节，标识了该 DataItem 是否有效。删除一个 DataItem，只需要简单地将其有效位设置为 0。DataSize 占用 2 字节，标识了后面 Data 的长度。",[10,130,131,132,135],{},"上层模块在获取到 DataItem 后，可以通过 ",[61,133,134],{"code":134},"data()"," 方法，该方法返回的数组是数据共享的，而不是拷贝实现的，所以使用了 SubArray。",[53,137,140],{"className":138,"code":139,"language":58,"meta":59},[56],"@Override\npublic SubArray data() {\n    return new SubArray(raw.raw, raw.start+OF_DATA, raw.end);\n}\n",[61,141,139],{"__ignoreMap":59},[10,143,144,145,148,149,152,153,156],{},"在上层模块试图对 DataItem 进行修改时，需要遵循一定的流程：在修改之前需要调用 ",[61,146,147],{"code":147},"before()"," 方法，想要撤销修改时，调用 ",[61,150,151],{"code":151},"unBefore()"," 方法，在修改完成后，调用 ",[61,154,155],{"code":155},"after()"," 方法。整个流程，主要是为了保存前相数据，并及时落日志。DM 会保证对 DataItem 的修改是原子性的。",[53,158,161],{"className":159,"code":160,"language":58,"meta":59},[56],"@Override\npublic void before() {\n    wLock.lock();\n    pg.setDirty(true);\n    System.arraycopy(raw.raw, raw.start, oldRaw, 0, oldRaw.length);\n}\n\n@Override\npublic void unBefore() {\n    System.arraycopy(oldRaw, 0, raw.raw, raw.start, oldRaw.length);\n    wLock.unlock();\n}\n\n@Override\npublic void after(long xid) {\n    dm.logDataItem(xid, this);\n    wLock.unlock();\n}\n",[61,162,160],{"__ignoreMap":59},[10,164,165,167],{},[61,166,155],{"code":155}," 方法，主要就是调用 dm 中的一个方法，对修改操作落日志，不赘述。",[10,169,170],{},"在使用完 DataItem 后，也应当及时调用 release() 方法，释放掉 DataItem 的缓存（由 DM 缓存 DataItem）。",[53,172,175],{"className":173,"code":174,"language":58,"meta":59},[56],"@Override\npublic void release() {\n    dm.releaseDataItem(this);\n}\n",[61,176,174],{"__ignoreMap":59},[34,178,180],{"id":179},"dm-的实现","DM 的实现",[10,182,183],{},"DataManager 是 DM 层直接对外提供方法的类，同时，也实现成 DataItem 对象的缓存。DataItem 存储的 key，是由页号和页内偏移组成的一个 8 字节无符号整数，页号和偏移各占 4 字节。",[10,185,186,187,190],{},"DataItem 缓存，",[61,188,189],{"code":189},"getForCache()","，只需要从 key 中解析出页号，从 pageCache 中获取到页面，再根据偏移，解析出 DataItem 即可：",[53,192,195],{"className":193,"code":194,"language":58,"meta":59},[56],"@Override\nprotected DataItem getForCache(long uid) throws Exception {\n    short offset = (short)(uid & ((1L \u003C\u003C 16) - 1));\n    uid >>>= 32;\n    int pgno = (int)(uid & ((1L \u003C\u003C 32) - 1));\n    Page pg = pc.getPage(pgno);\n    return DataItem.parseDataItem(pg, offset, this);\n}\n",[61,196,194],{"__ignoreMap":59},[10,198,199],{},"DataItem 缓存释放，需要将 DataItem 写回数据源，由于对文件的读写是以页为单位进行的，只需要将 DataItem 所在的页 release 即可：",[53,201,204],{"className":202,"code":203,"language":58,"meta":59},[56],"@Override\nprotected void releaseForCache(DataItem di) {\n    di.page().release();\n}\n",[61,205,203],{"__ignoreMap":59},[10,207,208],{},"从已有文件创建 DataManager 和从空文件创建 DataManager 的流程稍有不同，除了 PageCache 和 Logger 的创建方式有所不同以外，从空文件创建首先需要对第一页进行初始化，而从已有文件创建，则是需要对第一页进行校验，来判断是否需要执行恢复流程。并重新对第一页生成随机字节。",[53,210,213],{"className":211,"code":212,"language":58,"meta":59},[56],"public static DataManager create(String path, long mem, TransactionManager tm) {\n    PageCache pc = PageCache.create(path, mem);\n    Logger lg = Logger.create(path);\n    DataManagerImpl dm = new DataManagerImpl(pc, lg, tm);\n    dm.initPageOne();\n    return dm;\n}\n\npublic static DataManager open(String path, long mem, TransactionManager tm) {\n    PageCache pc = PageCache.open(path, mem);\n    Logger lg = Logger.open(path);\n    DataManagerImpl dm = new DataManagerImpl(pc, lg, tm);\n    if(!dm.loadCheckPageOne()) {\n        Recover.recover(tm, lg, pc);\n    }\n    dm.fillPageIndex();\n    PageOne.setVcOpen(dm.pageOne);\n    dm.pc.flushPage(dm.pageOne);\n    return dm;\n}\n",[61,214,212],{"__ignoreMap":59},[10,216,217],{},"其中，初始化第一页，和校验第一页，基本都是调用 PageOne 类中的方法实现的：",[53,219,222],{"className":220,"code":221,"language":58,"meta":59},[56],"\u002F\u002F 在创建文件时初始化 PageOne\nvoid initPageOne() {\n    int pgno = pc.newPage(PageOne.InitRaw());\n    assert pgno == 1;\n    try {\n        pageOne = pc.getPage(pgno);\n    } catch (Exception e) {\n        Panic.panic(e);\n    }\n    pc.flushPage(pageOne);\n}\n\n\u002F\u002F 在打开已有文件时时读入 PageOne，并验证正确性\nboolean loadCheckPageOne() {\n    try {\n        pageOne = pc.getPage(1);\n    } catch (Exception e) {\n        Panic.panic(e);\n    }\n    return PageOne.checkVc(pageOne);\n}\n",[61,223,221],{"__ignoreMap":59},[10,225,226,227,26,230,233],{},"DM 层提供了三个功能供上层使用，分别是读、插入和修改。修改是通过读出的 DataItem 实现的，于是 DataManager 只需要提供 ",[61,228,229],{"code":229},"read()",[61,231,232],{"code":232},"insert()"," 方法。",[10,235,236,238],{},[61,237,229],{"code":229}," 根据 UID 从缓存中获取 DataItem，并校验有效位：",[53,240,243],{"className":241,"code":242,"language":58,"meta":59},[56],"@Override\npublic DataItem read(long uid) throws Exception {\n    DataItemImpl di = (DataItemImpl)super.get(uid);\n    if(!di.isValid()) {\n        di.release();\n        return null;\n    }\n    return di;\n}\n",[61,244,242],{"__ignoreMap":59},[10,246,247,249],{},[61,248,232],{"code":232}," 方法，在 pageIndex 中获取一个足以存储插入内容的页面的页号，获取页面后，首先需要写入插入日志，接着才可以通过 pageX 插入数据，并返回插入位置的偏移。最后需要将页面信息重新插入 pageIndex。",[53,251,254],{"className":252,"code":253,"language":58,"meta":59},[56],"@Override\npublic long insert(long xid, byte[] data) throws Exception {\n    byte[] raw = DataItem.wrapDataItemRaw(data);\n    if(raw.length > PageX.MAX_FREE_SPACE) {\n        throw Error.DataTooLargeException;\n    }\n\n    \u002F\u002F 尝试获取可用页\n    PageInfo pi = null;\n    for(int i = 0; i \u003C 5; i ++) {\n        pi = pIndex.select(raw.length);\n        if (pi != null) {\n            break;\n        } else {\n            int newPgno = pc.newPage(PageX.initRaw());\n            pIndex.add(newPgno, PageX.MAX_FREE_SPACE);\n        }\n    }\n    if(pi == null) {\n        throw Error.DatabaseBusyException;\n    }\n\n    Page pg = null;\n    int freeSpace = 0;\n    try {\n        pg = pc.getPage(pi.pgno);\n        \u002F\u002F 首先做日志\n        byte[] log = Recover.insertLog(xid, pg, raw);\n        logger.log(log);\n        \u002F\u002F 再执行插入操作\n        short offset = PageX.insert(pg, raw);\n\n        pg.release();\n        return Types.addressToUid(pi.pgno, offset);\n\n    } finally {\n        \u002F\u002F 将取出的 pg 重新插入 pIndex\n        if(pg != null) {\n            pIndex.add(pi.pgno, PageX.getFreeSpace(pg));\n        } else {\n            pIndex.add(pi.pgno, freeSpace);\n        }\n    }\n}\n",[61,255,253],{"__ignoreMap":59},[10,257,258],{},"DataManager 正常关闭时，需要执行缓存和日志的关闭流程，不要忘了设置第一页的字节校验：",[53,260,263],{"className":261,"code":262,"language":58,"meta":59},[56],"@Override\npublic void close() {\n    super.close();\n    logger.close();\n\n    PageOne.setVcClose(pageOne);\n    pageOne.release();\n    pc.close();\n}\n",[61,264,262],{"__ignoreMap":59},[10,266,267],{},"以上，DM 层完结。",[10,269,270],{},"今天是 2021 年 12 月 11 日，A-SOUL 一周年直播。祝 A-SOUL 一周年快乐！还会有两周年、三周年、十周年！鸟巢见！！！",[10,272,273],{},[274,275],"img",{"alt":59,"src":276},"https:\u002F\u002Fblog-img.774352199.xyz\u002F2025\u002Fbee2e73291a2ecde2667bb41f2e2c5b6.jpg",[10,278,279],{},"我们是，ASOUL！",{"title":59,"searchDepth":281,"depth":281,"links":282},4,[283,285,286,287],{"id":36,"depth":284,"text":36},3,{"id":42,"depth":284,"text":42},{"id":98,"depth":284,"text":99},{"id":179,"depth":284,"text":180},[289],"项目","2021-12-11 15:16:00","页面索引是 DM 层的重要组成部分，通过缓存每一页的空闲空间来优化插入操作。这一机制使得上层模块能够迅速定位合适的页面，避免了冗长的搜索过程，提高了数据处理的效率。在实现方面，页面索引与数据项（DataItem）的抽象紧密结合，为数据库的高效运行提供了支撑。",false,"md",null,{"slots":296},{},true,"\u002Fprojects\u002Fmydb\u002Fmydb5",{"text":300,"minutes":301,"time":302,"words":303},"9 min read",8.9,534000,1780,{"title":5,"description":291},{"loc":298},"posts\u002Fprojects\u002Fmydb\u002Fmydb5",[58,308],"mydb","tech","BASj1WvjEq3nfGfZS7CFQsSrKwRaUuqVMYv77fFijjY",[312,317],{"title":313,"path":314,"stem":315,"date":316,"type":309,"children":-1},"MYDB 4. 日志文件与恢复策略","\u002Fprojects\u002Fmydb\u002Fmydb4","posts\u002Fprojects\u002Fmydb\u002Fmydb4","2021-12-08 22:55:00",{"title":318,"path":319,"stem":320,"date":321,"type":309,"children":-1},"MYDB 6. 记录的版本与事务隔离","\u002Fprojects\u002Fmydb\u002Fmydb6","posts\u002Fprojects\u002Fmydb\u002Fmydb6","2021-12-18 14:58:00",1787554445808]