Benchmark Results

Computational Resources

All benchmarks were performed on the same machine to ensure consistent comparisons.

• CPU: Apple M4
• Number of Cores: 10
• Memory: 16GB RAM
• Operating System: macOS Sequoia 15.5
• Julia Version: 1.10.9
• R Version: 4.4.2

Benchmarking Methods

To assess the efficiency of MetaCommunityMetrics compared to equivalent R implementations, we benchmark our functions against their R counterparts, focusing on execution time and memory usage. The following tables summarize the benchmark results based on 100 samples each.

We tested using datasets of three sizes:

• Small (5,325 observations)
• Medium (26,676 observations)
• Large (53,352 observations)

The large dataset is the sample data included with MetaCommunityMetrics, accessible via load_sample_data(). The small and medium datasets can be accessed here.

Each function was benchmarked using 100 samples in both BenchmarkTools.jl in Julia and bench::mark() in R to ensure robust statistical sampling. For memory usage comparisons:

• In Julia, we report the memory estimate from BenchmarkTools.jl, which measures bytes allocated during the trial with minimum elapsed time
• In R, we report the mem_alloc metric from bench::mark(), which tracks R heap allocations

According to documentation, the Julia metric measures total memory allocation during execution, while the R metric specifically tracks heap allocations within the R runtime, excluding "memory allocated outside the R heap, e.g., by malloc() or new directly." Due to differences in language implementation and measurement methodology, direct numerical comparisons between languages should be interpreted with caution.

Speedup Summary

Below is a plot showing the speedup of all benchmarked functions across the three datasets (small, medium and large). Speedup is calculated as the R median execution time divided by the Julia median execution time.

Benchmark Results in Details

All times are in millisecond (ms), and memory is in mebibytes (MiB). All values are rounded up to 4 decimal places.

Median Execution time and Speedup Values

95% confidence interval is reported.

Memory Usage

Benchmarked using Large Dataset

Benchmarked using Medium Dataset

Benchmarked using Small Dataset

Remarks

• For DNCI_multigroup_result, 100 permutations are used in both the Julia and R implementation, and parallelComputing was set to be TRUE when benchmarking DNCImper:::DNCI_multigroup() in R.

The Scripts Used for Benchmarking

• Julia
• R

Packages used for benchmarking

• bench
• BenchmarkTools.jl