Back to projects
2026Ongoing

miniDB — Relational Database Engine

A relational database engine built from scratch in Go featuring ACID transactions, B-Tree indexes, SQL query planning, recovery management, and a custom storage engine.

Source Code

Technology Stack

GoSQLB-TreeHash IndexingWrite-Ahead LoggingConcurrency ControlStorage EnginesQuery Planning

Overview

Core Features

  • •Custom page-based storage engine with block-oriented disk management.
  • •Buffer pool manager with configurable replacement strategies.
  • •ACID-compliant transaction system supporting commit, rollback, logging, and recovery.
  • •Concurrency control through lock management and transaction isolation primitives.
  • •SQL parser and query planner supporting SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, CREATE VIEW, and CREATE INDEX.
  • •B-Tree and Hash index implementations for efficient lookups.
  • •Query execution engine supporting joins, projections, filtering, sorting, aggregation, GROUP BY, HAVING, and ORDER BY.
  • •Metadata catalog managing schemas, tables, views, statistics, and indexes.
  • •Native database/sql driver integration allowing DropDB to be used like a standard Go database.

System Architecture

miniDB architecture
  • •Storage Layer: Page-based file manager handling block allocation, persistence, and disk I/O.
  • •Buffer Manager: Memory-resident page cache with pin/unpin semantics and replacement strategies.
  • •Transaction Layer: Write-ahead logging, checkpoints, rollback, recovery, and concurrency control.
  • •Query Layer: Lexer, parser, optimizer, planner, and execution operators.
  • •Index Layer: B-Tree and Hash indexes for accelerating query execution.
  • •Driver Layer: SQL driver exposing DropDB through Go's database/sql interface.

Key Challenges

  • •Designing a page-oriented storage engine with efficient record layouts.
  • •Implementing crash-safe transaction recovery using write-ahead logging.
  • •Building B-Tree index split and traversal algorithms.
  • •Managing concurrent transactions through lock coordination.
  • •Translating SQL queries into executable query plans.

Key Learnings

  • •Database internals including storage engines, buffer pools, and recovery systems.
  • •Query planning and execution pipelines.
  • •Transaction isolation and concurrency control mechanisms.
  • •Index design tradeoffs between B-Tree and Hash structures.
  • •Low-level systems programming in Go.

Impact

  • •Built a feature-rich relational database engine comprising storage, indexing, query processing, and transaction management subsystems.
  • •Demonstrated end-to-end understanding of database architecture beyond application-level backend development.