Back to projects
2026Ongoing

miniDB — Relational Database Engine

A relational database engine built from scratch in Go featuring ACID transactions, B-Tree indexes, SQL query planning, recovery management, and a custom storage engine.

Source Code

Technology Stack

GoSQLB-TreeHash IndexingWrite-Ahead LoggingConcurrency ControlStorage EnginesQuery Planning

Overview

miniDB is a relational database engine built from first principles to explore how modern databases such as PostgreSQL and MySQL work internally. The project implements the complete database pipeline including storage management, indexing, SQL parsing, query planning, transaction processing, concurrency control, and crash recovery.

Core Features

  • •Custom page-based storage engine with block-oriented disk management.
  • •Buffer pool manager with configurable replacement strategies.
  • •ACID-compliant transaction system supporting commit, rollback, logging, and recovery.
  • •Concurrency control through lock management and transaction isolation primitives.
  • •SQL parser and query planner supporting SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, CREATE VIEW, and CREATE INDEX.
  • •B-Tree and Hash index implementations for efficient lookups.
  • •Query execution engine supporting joins, projections, filtering, sorting, aggregation, GROUP BY, HAVING, and ORDER BY.
  • •Metadata catalog managing schemas, tables, views, statistics, and indexes.
  • •Native database/sql driver integration allowing DropDB to be used like a standard Go database.

System Architecture

miniDB architecture
  • •Storage Layer: Page-based file manager handling block allocation, persistence, and disk I/O.
  • •Buffer Manager: Memory-resident page cache with pin/unpin semantics and replacement strategies.
  • •Transaction Layer: Write-ahead logging, checkpoints, rollback, recovery, and concurrency control.
  • •Query Layer: Lexer, parser, optimizer, planner, and execution operators.
  • •Index Layer: B-Tree and Hash indexes for accelerating query execution.
  • •Driver Layer: SQL driver exposing DropDB through Go's database/sql interface.

Key Challenges

  • •Designing a page-oriented storage engine with efficient record layouts.
  • •Implementing crash-safe transaction recovery using write-ahead logging.
  • •Building B-Tree index split and traversal algorithms.
  • •Managing concurrent transactions through lock coordination.
  • •Translating SQL queries into executable query plans.

Key Learnings

  • •Database internals including storage engines, buffer pools, and recovery systems.
  • •Query planning and execution pipelines.
  • •Transaction isolation and concurrency control mechanisms.
  • •Index design tradeoffs between B-Tree and Hash structures.
  • •Low-level systems programming in Go.

Impact

  • •Built a feature-rich relational database engine comprising storage, indexing, query processing, and transaction management subsystems.
  • •Demonstrated end-to-end understanding of database architecture beyond application-level backend development.