JK
>LOADING // LOADING
INITIALIZING VIEW
Completedgoneo4jmemgraphtigergraphdocker
Graph Database Benchmark
Go CLI tool that benchmarks four graph databases (CognoDB, Neo4j, Memgraph, TigerGraph) using the SNAP soc-Pokec social network dataset with 49,683 nodes and 100,000 relationships. Uses adapter pattern for database abstraction with 10 CLI entry points and interactive HTML chart generation.
Problem & Solution
Provides an apples-to-apples comparison of graph database performance under controlled conditions, eliminating variables like dataset size, query design, and client implementation. Targets academic researchers and engineers evaluating graph database options under constrained (free-tier) resources.
Features
- Standardized benchmark queries across 4 graph databases (CognoDB, Neo4j, Memgraph, TigerGraph)
- 7 query workloads: point lookup, indexed lookup, 1/2/3-hop traversals, aggregation, write
- Sequential and concurrent (multi-goroutine) execution modes
- Cold-start latency measurement (iteration 0 tagging)
- Statistical aggregation: p50, p95, mean, stddev, min, max
- Interactive HTML bar chart generation via go-echarts
- Idempotent data loading via MERGE (upsert) operations
- Panic recovery via safeExec() prevents benchmark crashes
Tech Stack
| Technology | Version | Purpose | Why Chosen |
|---|---|---|---|
| Go | 1.25.1 | Primary language | Strong concurrency primitives, fast compilation, good driver ecosystem |
| neo4j-go-driver/v6 | v6.2.0 | Bolt protocol client for CognoDB, Neo4j, Memgraph | Official driver, supports Bolt 5.x, parameterized queries |
| go-echarts/v2 | v2.7.2 | HTML chart generation | Go-native, generates standalone HTML |
| godotenv | v1.5.1 | Environment configuration | Standard .env loading |
| net/http | stdlib | REST client for TigerGraph | No external dependency needed |
Architecture
Benchmark Engine
- Adapter pattern with common GraphDatabase interface (16 methods)
- 5 database adapters (4 active, FalkorDB disabled due to OOM)
- Sequential workload executor with warm-up and measured iterations
- Concurrent workload executor with N goroutines and sync.WaitGroup
- Factory function for adapter creation from configuration
CLI Pipeline
- 10 CLI entry points: benchmark, captest, charts, cleanup, dataset, load, runall, summary, verify, verify-load
- JSONL append-only result output for crash resilience
- CSV summary aggregation with percentile calculation
- HTML chart generation via go-echarts
- Pipeline automation via runall (verify → load → benchmark → report)
Project Structure
cmd
benchmark
captest
charts
cleanup
dataset
load
runall
summary
verify
verify-load
internal
benchmark
concurrent.go
measurement.go
runner.go
safe.go
database
cognodb
memgraph
neo4j
tigergraph
types
factory.go
dataset
parser.go
API Endpoints
Query Workloads (port 0)
| Method | Route | Purpose |
|---|---|---|
| Cypher | MATCH (n:Person {id: $id}) RETURN n.id AS id | Point lookup — single node by ID |
| Cypher | MATCH (n:Person {region: $val}) RETURN n.id AS id LIMIT 10 | Indexed lookup — nodes by region |
| Cypher | MATCH (n:Person {id: $id})-[:FRIEND]->(m:Person) RETURN collect(m.id) | 1-hop traversal — direct friends |
| Cypher | MATCH (n:Person {id: $id})-[:FRIEND]->()-[:FRIEND]->(m:Person) RETURN collect(DISTINCT m.id) | 2-hop traversal — friends of friends |
| Cypher | MATCH (n:Person {id: $id})-[:FRIEND]->()-[:FRIEND]->()-[:FRIEND]->(m:Person) RETURN collect(DISTINCT m.id) | 3-hop traversal — extended network |
| Cypher | MATCH (n:Person) RETURN n.region AS region, count(n) AS cnt ORDER BY cnt DESC LIMIT 10 | Aggregation — region distribution |
| Cypher | CREATE (n:Person {id: $id, region: $region, ...}) | Write — create new node |
Database Design
Person (node label)
| Column | Type | Notes |
|---|---|---|
| idPK | STRING | Unique constraint, indexed |
| region | STRING | Indexed |
| gender | STRING | Not indexed |
| age | STRING | Indexed (CognoDB, Neo4j, Memgraph only) |
| completion_percentage | STRING | Not indexed |
49,683 nodes extracted from SNAP soc-Pokec dataset. All properties are strings.
FRIEND (relationship type)
| Column | Type | Notes |
|---|---|---|
| source_id | STRING | Person.id (from) |
| target_id | STRING | Person.id (to) |
100,000 directed relationships. No properties on edges. Many-to-many cardinality.
Directed graph: Person --[FRIEND]--> Person. Many-to-many via graph relationships (no junction table). Relationships are first-class citizens in the graph model.
Key Design Decisions
Single Bolt Driver for 3 Databases
Decision: Use neo4j-go-driver/v6 for CognoDB, Neo4j, and Memgraph
Why: All three support Bolt protocol; Cypher is logically equivalent
Benefits: Code reuse, consistent behavior across databases, adapter pattern isolates DB-specific behavior
Trade-offs: Cannot test database-specific optimizations; Memgraph has its own Go driver
MaxPoolSize = 1
Decision: Single connection per benchmark run despite constant defining 10
Why: Measure single-connection latency; avoid connection pool effects
Benefits: Clean latency measurement without pool contention
Trade-offs: Doesn't test connection pooling behavior
JSONL for Raw Results
Decision: One JSON object per line, append-only output
Why: Stream results without loading all into memory; easy to append across runs
Benefits: Resilient to crashes (partial results preserved)
Trade-offs: Requires full file scan for aggregation
No Retries on Measured Queries
Decision: Zero retries for benchmark queries to capture true latency
Why: Network blips and failures are part of real-world performance
Benefits: Honest performance measurement including failure modes
Trade-offs: Network blips cause false negatives
Goroutine-Per-Client Concurrency
Decision: Each concurrent client creates its own database connection
Why: Simulates real-world multi-client scenarios; avoids shared connection contention
Benefits: Realistic concurrency testing with independent connections
Trade-offs: Higher resource usage; connection limits become the bottleneck
MERGE for Idempotent Node Loading
Decision: Use MERGE (upsert) instead of CREATE for node loading
Why: Makes loading idempotent — running multiple times doesn't create duplicates
Benefits: Safe re-runs, consistent dataset state
Trade-offs: MERGE is slightly slower than CREATE due to existence checks
Testing
None (no _test.go files)•0 unit tests
What's Tested
- Capability test matrix (cmd/captest/) — tests all 8 workload types against all databases with small 10-node graph
- Connection verification (cmd/verify/) — tests basic connectivity and parameterized queries
- Data integrity verification (cmd/verify-load/) — verifies node/relationship counts (49,683 nodes, 100,000 relationships)
Missing Coverage
- No unit tests for individual functions (_test.go files do not exist)
- No tests for statistical functions (percentile, mean, stddev calculation)
- No tests for CSV parsing edge cases (empty datasets, malformed CSVs)
- No tests for concurrent behavior
- No tests for chart generation correctness
- No test framework installed (no testify, gomock)
Key Challenges
- FalkorDB OOM — free tier 100MB RAM limit exceeded by 49K-node dataset, entire adapter disabled
- Protocol asymmetry — Bolt (binary) vs REST (HTTP/JSON) confounds direct latency comparison
- Resource asymmetry — CognoDB 0.5 vCPU/256MB vs competitors with 2-8x more resources
- Free-tier variability — shared resources, auto-pause, time limits affect reproducibility
- Ensuring fair comparison across different graph database architectures
Key Learnings
- Adapter pattern for database-agnostic benchmarking with clean interface separation
- Goroutine-per-client concurrency model for realistic multi-client simulation
- JSONL append-only output for crash-resilient result persistence
- Cold-start measurement technique (iteration 0 tagging after warm-up)
- Numerical stability in statistical aggregation (percentile calculation)