Benchmark Wydajności Baz Grafowych
Pakiet testów analizujący opóźnienia, zużycie pamięci i przepustowość między bazami grafowymi a relacyjnymi.
Problem i rozwiązanie
Wybór optymalnej bazy danych do złożonych sieci społecznościowych i systemów rekomendacji.
Funkcje
- Zautomatyzowane testy przejść grafowych na różnych głębokościach
- Porównanie percentyli opóźnień (p50, p95, p99)
- Szczegółowy monitoring zużycia pamięci i procesora
- W pełni powtarzalne środowisko testowe w Dockerze
Stos technologiczny
| Technologia | Wersja | Cel | Dlaczego wybrano |
|---|---|---|---|
| Go | 1.25.1 | Primary language | Strong concurrency primitives, fast compilation, good driver ecosystem |
| neo4j-go-driver/v6 | v6.2.0 | Bolt protocol client for CognoDB, Neo4j, Memgraph | Official driver, supports Bolt 5.x, parameterized queries |
| go-echarts/v2 | v2.7.2 | HTML chart generation | Go-native, generates standalone HTML |
| godotenv | v1.5.1 | Environment configuration | Standard .env loading |
| net/http | stdlib | REST client for TigerGraph | No external dependency needed |
Architektura
Benchmark Engine
- Adapter pattern with common GraphDatabase interface (16 methods)
- 5 database adapters (4 active, FalkorDB disabled due to OOM)
- Sequential workload executor with warm-up and measured iterations
- Concurrent workload executor with N goroutines and sync.WaitGroup
- Factory function for adapter creation from configuration
CLI Pipeline
- 10 CLI entry points: benchmark, captest, charts, cleanup, dataset, load, runall, summary, verify, verify-load
- JSONL append-only result output for crash resilience
- CSV summary aggregation with percentile calculation
- HTML chart generation via go-echarts
- Pipeline automation via runall (verify → load → benchmark → report)
Struktura projektu
Punkty końcowe API
Query Workloads (port 0)
| Metoda | Trasa | Cel |
|---|---|---|
| Cypher | MATCH (n:Person {id: $id}) RETURN n.id AS id | Point lookup — single node by ID |
| Cypher | MATCH (n:Person {region: $val}) RETURN n.id AS id LIMIT 10 | Indexed lookup — nodes by region |
| Cypher | MATCH (n:Person {id: $id})-[:FRIEND]->(m:Person) RETURN collect(m.id) | 1-hop traversal — direct friends |
| Cypher | MATCH (n:Person {id: $id})-[:FRIEND]->()-[:FRIEND]->(m:Person) RETURN collect(DISTINCT m.id) | 2-hop traversal — friends of friends |
| Cypher | MATCH (n:Person {id: $id})-[:FRIEND]->()-[:FRIEND]->()-[:FRIEND]->(m:Person) RETURN collect(DISTINCT m.id) | 3-hop traversal — extended network |
| Cypher | MATCH (n:Person) RETURN n.region AS region, count(n) AS cnt ORDER BY cnt DESC LIMIT 10 | Aggregation — region distribution |
| Cypher | CREATE (n:Person {id: $id, region: $region, ...}) | Write — create new node |
Projekt bazy danych
Person (node label)
| Kolumna | Typ | Uwagi |
|---|---|---|
| idPK | STRING | Unique constraint, indexed |
| region | STRING | Indexed |
| gender | STRING | Not indexed |
| age | STRING | Indexed (CognoDB, Neo4j, Memgraph only) |
| completion_percentage | STRING | Not indexed |
49,683 nodes extracted from SNAP soc-Pokec dataset. All properties are strings.
FRIEND (relationship type)
| Kolumna | Typ | Uwagi |
|---|---|---|
| source_id | STRING | Person.id (from) |
| target_id | STRING | Person.id (to) |
100,000 directed relationships. No properties on edges. Many-to-many cardinality.
Directed graph: Person --[FRIEND]--> Person. Many-to-many via graph relationships (no junction table). Relationships are first-class citizens in the graph model.
Kluczowe decyzje projektowe
Single Bolt Driver for 3 Databases
Decyzja: Use neo4j-go-driver/v6 for CognoDB, Neo4j, and Memgraph
Dlaczego: All three support Bolt protocol; Cypher is logically equivalent
Korzyści: Code reuse, consistent behavior across databases, adapter pattern isolates DB-specific behavior
Kompromisy: Cannot test database-specific optimizations; Memgraph has its own Go driver
MaxPoolSize = 1
Decyzja: Single connection per benchmark run despite constant defining 10
Dlaczego: Measure single-connection latency; avoid connection pool effects
Korzyści: Clean latency measurement without pool contention
Kompromisy: Doesn't test connection pooling behavior
JSONL for Raw Results
Decyzja: One JSON object per line, append-only output
Dlaczego: Stream results without loading all into memory; easy to append across runs
Korzyści: Resilient to crashes (partial results preserved)
Kompromisy: Requires full file scan for aggregation
No Retries on Measured Queries
Decyzja: Zero retries for benchmark queries to capture true latency
Dlaczego: Network blips and failures are part of real-world performance
Korzyści: Honest performance measurement including failure modes
Kompromisy: Network blips cause false negatives
Goroutine-Per-Client Concurrency
Decyzja: Each concurrent client creates its own database connection
Dlaczego: Simulates real-world multi-client scenarios; avoids shared connection contention
Korzyści: Realistic concurrency testing with independent connections
Kompromisy: Higher resource usage; connection limits become the bottleneck
MERGE for Idempotent Node Loading
Decyzja: Use MERGE (upsert) instead of CREATE for node loading
Dlaczego: Makes loading idempotent — running multiple times doesn't create duplicates
Korzyści: Safe re-runs, consistent dataset state
Kompromisy: MERGE is slightly slower than CREATE due to existence checks
Testy
Co jest testowane
- Capability test matrix (cmd/captest/) — tests all 8 workload types against all databases with small 10-node graph
- Connection verification (cmd/verify/) — tests basic connectivity and parameterized queries
- Data integrity verification (cmd/verify-load/) — verifies node/relationship counts (49,683 nodes, 100,000 relationships)
Brakujące pokrycie
- No unit tests for individual functions (_test.go files do not exist)
- No tests for statistical functions (percentile, mean, stddev calculation)
- No tests for CSV parsing edge cases (empty datasets, malformed CSVs)
- No tests for concurrent behavior
- No tests for chart generation correctness
- No test framework installed (no testify, gomock)
Kluczowe wyzwania
- FalkorDB OOM — free tier 100MB RAM limit exceeded by 49K-node dataset, entire adapter disabled
- Protocol asymmetry — Bolt (binary) vs REST (HTTP/JSON) confounds direct latency comparison
- Resource asymmetry — CognoDB 0.5 vCPU/256MB vs competitors with 2-8x more resources
- Free-tier variability — shared resources, auto-pause, time limits affect reproducibility
- Ensuring fair comparison across different graph database architectures
Kluczowe wnioski
- Adapter pattern for database-agnostic benchmarking with clean interface separation
- Goroutine-per-client concurrency model for realistic multi-client simulation
- JSONL append-only output for crash-resilient result persistence
- Cold-start measurement technique (iteration 0 tagging after warm-up)
- Numerical stability in statistical aggregation (percentile calculation)