Benchmarking Results

Computational Resources

All benchmarks were performed on the same machine to ensure consistent comparisons.

  • CPU: Apple M4
  • Number of Cores: 10
  • Memory: 16GB RAM
  • Operating System: macOS Sequoia 15.6
  • Julia Version: 1.10.10
  • R Version: 4.4.2

Benchmarking Methods

To assess the efficiency of MetaCommunityMetrics compared to equivalent R implementations, we benchmark our functions against their R counterparts, focusing on execution time and memory usage. The following tables summarize the benchmark results based on 100 samples each.

We tested using datasets of three sizes:

  • Large (53,352 observations)
  • Medium (26,676 observations, 50% of the size of the large dataset)
  • Small (5,325 observations, 10% of the size of the large dataset)

The large dataset is the sample data included with MetaCommunityMetrics, accessible via load_sample_data(). The small and medium datasets can be accessed here.

Each function was benchmarked using 100 samples in both BenchmarkTools.jl in Julia and bench::mark() in R to ensure robust statistical sampling. For memory usage comparisons:

  • In Julia, we report the memory estimate from BenchmarkTools.jl, which measures bytes allocated for a given expression per sample.

  • In R, we report the mem_alloc metric from bench::mark(), which tracks R heap allocation per sample.

According to documentation, the Julia metric measures total memory allocation during execution, while the R metric specifically tracks heap allocations within the R runtime, excluding "memory allocated outside the R heap, e.g., by malloc() or new directly." Due to differences in language implementation and measurement methodology, direct numerical comparisons between languages should be interpreted with caution.

Speedup Summary

Below is a plot showing the speedup of all benchmarked functions across the three datasets (small, medium and large). Speedup is calculated as the R median execution time divided by the Julia median execution time. Speedup Plot

Benchmarking Results in Details

All times are in millisecond (ms), and memory is in mebibytes (MiB). All values are rounded up to 4 decimal places.

Median Execution time and Speedup Values

95% confidence interval is reported.

TestCaseDataSizeJuliaRSpeedupLower CIUpper CI
Beta Diversity (Abundance, quant=true)Large0.12442.147117.260216.607117.7147
Beta Diversity (Abundance, quant=true)Medium0.07781.373217.651817.376818.1861
Beta Diversity (Abundance, quant=true)Small0.03940.917923.300022.056125.3414
Beta Diversity (Abundance, quant=false)Large0.01460.199713.693413.235816.8835
Beta Diversity (Abundance, quant=false)Medium0.01040.213820.522619.668322.2222
Beta Diversity (Abundance, quant=false)Small0.00540.241444.572540.847247.8959
Beta Diversity (Presence, quant=false)Large0.01390.196214.165313.582814.7816
Beta Diversity (Presence, quant=false)Medium0.01130.199417.596817.112718.1192
Beta Diversity (Presence, quant=false)Small0.00570.251344.186339.511146.2503
Spatial Beta Diversity (Abundance, quant=true)Large2.16889.12344.20684.13874.3877
Spatial Beta Diversity (Abundance, quant=true)Medium1.47259.50226.45296.03586.5497
Spatial Beta Diversity (Abundance, quant=true)Small1.58399.49875.99715.75056.6586
Spatial Beta Diversity (Abundance, quant=false)Large1.94447.31793.76363.61243.8332
Spatial Beta Diversity (Abundance, quant=false)Medium1.66187.19574.33014.22454.4562
Spatial Beta Diversity (Abundance, quant=false)Small1.52076.91094.54474.39664.8593
Spatial Beta Diversity (Presence, quant=false)Large2.04198.03883.93703.84494.0554
Spatial Beta Diversity (Presence, quant=false)Medium1.70677.32844.29394.18924.4127
Spatial Beta Diversity (Presence, quant=false)Small1.47237.00794.75994.57375.1147
Temporal Beta Diversity (Abundance, quant=true)Large5.309653.945510.160010.078010.2358
Temporal Beta Diversity (Abundance, quant=true)Medium5.301553.353110.06389.860010.2946
Temporal Beta Diversity (Abundance, quant=true)Small4.743153.373211.252810.798411.5056
Temporal Beta Diversity (Abundance, quant=false)Large2.65489.56023.60123.54423.6613
Temporal Beta Diversity (Abundance, quant=false)Medium2.23959.44594.21794.08054.3567
Temporal Beta Diversity (Abundance, quant=false)Small2.21699.27614.18434.00344.3031
Temporal Beta Diversity (Presence, quant=false)Large2.687710.77954.01063.84444.1896
Temporal Beta Diversity (Presence, quant=false)Medium2.23639.78014.37344.24464.5393
Temporal Beta Diversity (Presence, quant=false)Small2.26829.22704.06803.96684.3038
Dispersal-niche continuum indexLarge588.765812574.846121.358021.215721.4509
Dispersal-niche continuum indexMedium505.202312268.218224.283824.187024.3857
Dispersal-niche continuum indexSmall122.36773140.294325.662825.432325.8697
Occupied Patches ProportionLarge0.84697.24558.55548.27378.8168
Occupied Patches ProportionMedium0.65047.311511.241610.868311.8014
Occupied Patches ProportionSmall0.36577.093419.397418.667620.1553
Variability MetricsLarge13.897998.99907.12337.01567.1537
Variability MetricsMedium7.913152.79036.67136.62246.7430
Variability MetricsSmall2.751814.26075.18245.12895.2272
Hypervolume EstimationLarge0.00350.03289.38318.422618.6374
Hypervolume EstimationMedium0.00320.02768.71278.58968.8098
Hypervolume EstimationSmall0.00290.02649.26439.06589.4181
Hypervolume DissimilarityLarge0.00640.114717.988517.625818.2162
Hypervolume DissimilarityMedium0.00560.118821.360521.164621.6136
Hypervolume DissimilaritySmall0.00500.185237.344934.640638.0277

Memory Usage

Benchmarked using Large Dataset

TestCaseJuliaR
Beta Diversity (Abundance, quant=true)0.43460.0566
Beta Diversity (Abundance, quant=false)0.13470.1252
Beta Diversity (Presence, quant=false)0.13470.1252
Spatial Beta Diversity (Abundance, quant=true)3.91853.1142
Spatial Beta Diversity (Abundance, quant=false)3.51732.6712
Spatial Beta Diversity (Presence, quant=false)3.51732.6712
Temporal Beta Diversity (Abundance, quant=true)16.883816.8863
Temporal Beta Diversity (Abundance, quant=false)5.65865.1752
Temporal Beta Diversity (Presence, quant=false)5.65865.1752
Dispersal-niche continuum index781.922071.9833
Occupied Patches Proportion1.90981.8857
Variability Metrics12.457360.2217
Hypervolume Estimation0.01220.0022
Hypervolume Dissimilarity0.01680.0145

Benchmarked using Medium Dataset

TestCaseJuliaR
Beta Diversity (Abundance, quant=true)0.24910.0357
Beta Diversity (Abundance, quant=false)0.10860.0798
Beta Diversity (Presence, quant=false)0.10860.0798
Spatial Beta Diversity (Abundance, quant=true)2.39012.2736
Spatial Beta Diversity (Abundance, quant=false)1.99151.8306
Spatial Beta Diversity (Presence, quant=false)1.99151.8306
Temporal Beta Diversity (Abundance, quant=true)15.277316.2509
Temporal Beta Diversity (Abundance, quant=false)4.13284.5397
Temporal Beta Diversity (Presence, quant=false)4.13284.5397
Dispersal-niche continuum index542.932759.2854
Occupied Patches Proportion0.99401.3994
Variability Metrics7.674632.5536
Hypervolume Estimation0.00850.0011
Hypervolume Dissimilarity0.01200.0077

Benchmarked using Small Dataset

TestCaseJuliaR
Beta Diversity (Abundance, quant=true)0.12250.0195
Beta Diversity (Abundance, quant=false)0.08830.0444
Beta Diversity (Presence, quant=false)0.08830.0444
Spatial Beta Diversity (Abundance, quant=true)1.13341.2126
Spatial Beta Diversity (Abundance, quant=false)0.76630.7696
Spatial Beta Diversity (Presence, quant=false)0.76630.7696
Temporal Beta Diversity (Abundance, quant=true)12.881115.4341
Temporal Beta Diversity (Abundance, quant=false)2.89373.7230
Temporal Beta Diversity (Presence, quant=false)2.89373.7230
Dispersal-niche continuum index192.578211.1101
Occupied Patches Proportion0.26120.4931
Variability Metrics3.848310.5805
Hypervolume Estimation0.00590.0003
Hypervolume Dissimilarity0.00820.0014

Datesets used for this benchmark

Large Dataset

53352×12 DataFrame
   Row │ Year   Month  Day    Sampling_date_order  plot   Species  Abundance  Presence  Latitude  Longitude  standardized_temperature  standardized_precipitation 
       │ Int64  Int64  Int64  Int64                Int64  String3  Int64      Int64     Float64   Float64    Float64                 Float64                  
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │  2010      1     16                    1      1  BA               0         0      35.0     -110.0                0.829467              -1.4024
     2 │  2010      1     16                    1      2  BA               0         0      35.0     -109.5               -1.12294               -0.0519895
     3 │  2010      1     16                    1      4  BA               0         0      35.0     -108.5               -0.409808              -0.803663
     4 │  2010      1     16                    1      8  BA               0         0      35.5     -109.5               -1.35913               -0.646369
     5 │  2010      1     16                    1      9  BA               0         0      35.5     -109.0                0.0822                 1.09485
   ⋮   │   ⋮      ⋮      ⋮             ⋮             ⋮       ⋮         ⋮         ⋮         ⋮          ⋮                ⋮                        ⋮
 53348 │  2023      3     21                  117      9  SH               0         0      35.5     -109.0               -0.571565              -0.836345
 53349 │  2023      3     21                  117     10  SH               0         0      35.5     -108.5               -2.33729               -0.398522
 53350 │  2023      3     21                  117     12  SH               1         1      35.5     -107.5                0.547169               1.03257
 53351 │  2023      3     21                  117     16  SH               0         0      36.0     -108.5               -0.815015               0.95971
 53352 │  2023      3     21                  117     23  SH               0         0      36.5     -108.0                0.48949               -1.59416
                                                                                                                                            53342 rows omitted

Medium Dataset

26676×12 DataFrame
   Row │ Year   Month  Day    Sampling_date_order  plot   Species  Abundance  Presence  Latitude  Longitude  standardized_temperature  standardized_precipitation 
       │ Int64  Int64  Int64  Int64                Int64  String3  Int64      Int64     Float64   Float64    Float64                 Float64                  
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │  2015      7     18                   56     12  PM               0         0      35.5     -107.5              -1.30965                  0.967859
     2 │  2016      8      6                   66     13  SF               0         0      36.0     -110.0               1.45692                  1.77253
     3 │  2017      2     25                   71     21  SF               0         0      36.5     -109.0              -1.50086                  0.993311
     4 │  2018      5     19                   82     16  PB               0         0      36.0     -108.5              -1.2202                   0.684295
   ⋮   │   ⋮      ⋮      ⋮             ⋮             ⋮       ⋮         ⋮         ⋮         ⋮          ⋮                ⋮                        ⋮
 26673 │  2010     11      6                    8     13  BA               0         0      36.0     -110.0              -1.05336                 -0.250324
 26674 │  2013      9     14                   36     12  NA               0         0      35.5     -107.5               0.213222                 0.12
 26675 │  2023      2     18                  116     12  DS               0         0      35.5     -107.5              -0.217475                 0.042571
 26676 │  2014     11     22                   49     13  PF               0         0      36.0     -110.0               0.613491                -1.17076
                                                                                                                                            26668 rows omitted

Small Dataset

5335×12 DataFrame
  Row │ Year   Month  Day    Sampling_date_order  plot   Species  Abundance  Presence  Latitude  Longitude  standardized_temperature  standardized_precipitation 
      │ Int64  Int64  Int64  Int64                Int64  String3  Int64      Int64     Float64   Float64    Float64                 Float64                  
──────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    1 │  2015      7     18                   56     12  PM               0         0      35.5     -107.5               -1.30965                 0.967859
    2 │  2016      8      6                   66     13  SF               0         0      36.0     -110.0                1.45692                 1.77253
    3 │  2017      2     25                   71     21  SF               0         0      36.5     -109.0               -1.50086                 0.993311
    4 │  2018      5     19                   82     16  PB               0         0      36.0     -108.5               -1.2202                  0.684295
  ⋮   │   ⋮      ⋮      ⋮             ⋮             ⋮       ⋮         ⋮         ⋮         ⋮          ⋮                ⋮                        ⋮
 5332 │  2013     11      2                   37     13  RF               0         0      36.0     -110.0                2.07639                 2.72841
 5333 │  2018     11     10                   88      6  PH               0         0      35.0     -107.5               -0.197006                0.842547
 5334 │  2018      3     24                   80      2  DS               0         0      35.0     -109.5               -0.920093               -0.276074
 5335 │  2014      4     26                   42      7  PH               0         0      35.5     -110.0                0.848755               -0.247144
                                                                                                                                            5327 rows omitted

Remarks

  • For DNCI_multigroup_result, 100 permutations per sample are used in both the Julia and R implementation, and parallelComputing was set to be TRUE when benchmarking DNCImper:::DNCI_multigroup() in R.

The Scripts Used for Benchmarking

Packages used for benchmarking