Project: https://github.com/CN-GuoZiyang/MYDB
Preface (Some Rambling)
Maybe I am addicted to reinventing wheels, or maybe I just felt I needed to brush up on database fundamentals. Either way, I finished this project in a little more than half a month, working from the end of the workday until midnight.
I do have a bit of history with databases. When my university offered its database systems course, I happened to be interning in Shenzhen. Online classes became a perfectly legitimate excuse to slack off at work, and I did just about everything except listen to the lectures. Operating systems was taught around the same time, but since I actually had some interest in OS, my grasp of it was not quite as shaky as my database knowledge.
The consequences caught up with me soon enough. In my second interview at ByteDance, the interviewer asked how much I knew about databases. In the spirit of being candid and clear, I said, “Nothing at all.” He then asked about redis, and I could only say, “Nothing about that either.” Fortunately, he did not hold it against me and passed me anyway. I do wonder whether those two “I don’t knows” tanked my interview feedback, though…
Once I started working, my team had little to do with databases, so I thought I had escaped databases and CRUD for good. Things changed surprisingly quickly: the neighboring asset management team was desperately short-staffed and asked the department to lend them people, so I was sent over to help. Asset management has fairly strict consistency requirements. I could hardly keep throwing every bit of data into redis and shrugging when something failed to go in…
What Got Me Started
One day, while browsing GitHub, I stumbled across @qw4990’s database project, NYADB2. It is a simple database written in Go, with an excellent layered design and code that is easy to read. Still fond of Java, I decided to write a Java database based on its overall architecture. I referred to this project for many implementation details along the way.
Rather embarrassingly for me, this was the author’s undergraduate hobby project. I suppose that is what being seriously good looks like. Runs away
RESPECT
Overall Structure
MYDB consists of a backend and a frontend, which communicate over sockets. The frontend (client) has a very simple job: read user input, send it to the backend for execution, print the returned result, and wait for the next input. The MYDB backend parses SQL and, if it is valid, attempts to execute it and returns the result. Excluding the parser, the backend is divided into five modules. Each has a defined responsibility and exposes methods through interfaces to the modules that depend on it. The five modules are:
- Transaction Manager (TM)
- Data Manager (DM)
- Version Manager (VM)
- Index Manager (IM)
- Table Manager (TBM)
Their dependencies look like this:
A topological sort of this dependency graph gives us an implementation order. In this tutorial, that order is TM -> DM -> VM -> IM -> TBM.
Here is what each module does:
- TM maintains transaction states in an XID file and exposes interfaces that other modules use to query a transaction’s state.
- DM directly manages the database’s DB file and log file. Its main responsibilities are: 1) managing and caching the pages of the DB file; 2) managing the log file so the database can recover from errors using the log; and 3) exposing the DB file as DataItems to higher-level modules and providing a cache for them.
- VM uses two-phase locking (2PL) to make schedules serializable and implements MVCC to eliminate blocking between reads and writes. It also implements two isolation levels.
- IM implements B+ tree indexes. By the way, where currently supports only indexed fields.
- TBM manages fields and tables. It also parses SQL statements and performs the corresponding operations on tables.
Development Environment and Example Run
I developed the project using WSL2 and JDK11. To run it on Windows, replace the paths in the launch arguments with Windows paths. Make sure you use JDK 11 or later; JDK 8 is not compatible (or find the incompatible methods yourself and replace them with compatible alternatives—there should only be a handful).
JDK 8 is now supported.
Almost every module and submodule has corresponding unit tests in the test folder. Please write plenty of unit tests yourself, too. Otherwise, when you finally run everything together, you will have no idea where the bugs came from.
Skipping unit tests feels great—until a bug turns everything into a dumpster fire. (
First, adjust the compilation version in pom.xml. If you import the project into an IDE, change its compilation version to match your JDK.
First, compile the source:
mvn compile
Next, create a database at /tmp/mydb:
mvn exec:java -Dexec.mainClass="top.guoziyang.mydb.backend.Launcher" -Dexec.args="-create /tmp/mydb"
Then start the database server with the default parameters:
mvn exec:java -Dexec.mainClass="top.guoziyang.mydb.backend.Launcher" -Dexec.args="-open /tmp/mydb"
The database server is now listening on port 9999 on your machine. Open another terminal and run the following command to start a client and connect to it:
mvn exec:java -Dexec.mainClass="top.guoziyang.mydb.client.Launcher"
This starts an interactive command line where you can enter SQL-like statements. Press Enter to send a statement to the server and print the result.
Here is an example:
