Benchmarking Results

Computational Resources

All benchmarks were performed on the same machine to ensure consistent comparisons.

• CPU: Apple M4
• Number of Cores: 10
• Memory: 16GB RAM
• Operating System: macOS Tahoe 26.5.1
• Julia Version: 1.12.6
• R Version: 4.4.2

Benchmarking Methods

To assess the efficiency of MetaCommunityMetrics compared to equivalent R implementations, we benchmark our functions against their R counterparts, focusing on execution time and memory usage. The following tables summarize the benchmark results based on 100 samples each.

We tested using datasets of three sizes:

• Large (53,352 observations)
• Medium (26,676 observations, 50% of the size of the large dataset)
• Small (5,325 observations, 10% of the size of the large dataset)

The large dataset is the sample data included with MetaCommunityMetrics, accessible via load_sample_data(). The small and medium datasets can be accessed here.

Each function was benchmarked using 100 samples in both BenchmarkTools.jl in Julia and bench::mark() in R to ensure robust statistical sampling. For memory usage comparisons:

• In Julia, we report the memory estimate from BenchmarkTools.jl, which measures bytes allocated for a given expression per sample.

• In R, we report the mem_alloc metric from bench::mark(), which tracks R heap allocation per sample.

According to documentation, the Julia metric measures total memory allocation during execution, while the R metric specifically tracks heap allocations within the R runtime, excluding "memory allocated outside the R heap, e.g., by malloc() or new directly." Due to differences in language implementation and measurement methodology, direct numerical comparisons between languages should be interpreted with caution.

Speedup Summary

Below is a plot showing the speedup of all benchmarked functions across the three datasets (small, medium and large). Speedup is calculated as the R median execution time divided by the Julia median execution time. Median speedup and its confidence interval of each comparison is shown. The grey dashed line shows where speedup equals to 1, that is both R and Julia's implementations require the same amount of time. The calcualation of beta diversity has two variants: quant = true (abundance data is used) and quant = false (occurence data is used).

Benchmarking Results in Details

All times are in millisecond (ms), and memory is in mebibytes (MiB). All values are rounded up to 4 decimal places.

Median Execution Time and Speedup Values

Bold text indicates the test cases with maximum and minimum speedup values. 95% confidence interval of the speedup is reported.

Memory Usage

Benchmarked using Large Dataset

Bold text indicates the test case with the biggest memory usage difference between Julia and R.

Benchmarked using Medium Dataset

The test case with the biggest memory usage difference between Julia and R is highligthed.

Benchmarked using Small Dataset

The test case with the biggest memory usage difference between Julia and R is highligthed.

Datasets used for this benchmark

Large Dataset

53352×12 DataFrame
   Row │ Year   Month  Day    Sampling_date_order  plot   Species  Abundance  Presence  Latitude  Longitude  standardized_temperature  standardized_precipitation 
       │ Int64  Int64  Int64  Int64                Int64  String3  Int64      Int64     Float64   Float64    Float64                 Float64                  
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │  2010      1     16                    1      1  BA               0         0      35.0     -110.0                0.829467              -1.4024
     2 │  2010      1     16                    1      2  BA               0         0      35.0     -109.5               -1.12294               -0.0519895
     3 │  2010      1     16                    1      4  BA               0         0      35.0     -108.5               -0.409808              -0.803663
     4 │  2010      1     16                    1      8  BA               0         0      35.5     -109.5               -1.35913               -0.646369
     5 │  2010      1     16                    1      9  BA               0         0      35.5     -109.0                0.0822                 1.09485
   ⋮   │   ⋮      ⋮      ⋮             ⋮             ⋮       ⋮         ⋮         ⋮         ⋮          ⋮                ⋮                        ⋮
 53348 │  2023      3     21                  117      9  SH               0         0      35.5     -109.0               -0.571565              -0.836345
 53349 │  2023      3     21                  117     10  SH               0         0      35.5     -108.5               -2.33729               -0.398522
 53350 │  2023      3     21                  117     12  SH               1         1      35.5     -107.5                0.547169               1.03257
 53351 │  2023      3     21                  117     16  SH               0         0      36.0     -108.5               -0.815015               0.95971
 53352 │  2023      3     21                  117     23  SH               0         0      36.5     -108.0                0.48949               -1.59416
                                                                                                                                            53342 rows omitted

Medium Dataset

26676×12 DataFrame
   Row │ Year   Month  Day    Sampling_date_order  plot   Species  Abundance  Presence  Latitude  Longitude  standardized_temperature  standardized_precipitation 
       │ Int64  Int64  Int64  Int64                Int64  String3  Int64      Int64     Float64   Float64    Float64                 Float64                  
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     1 │  2015      7     18                   56     12  PM               0         0      35.5     -107.5              -1.30965                  0.967859
     2 │  2016      8      6                   66     13  SF               0         0      36.0     -110.0               1.45692                  1.77253
     3 │  2017      2     25                   71     21  SF               0         0      36.5     -109.0              -1.50086                  0.993311
     4 │  2018      5     19                   82     16  PB               0         0      36.0     -108.5              -1.2202                   0.684295
   ⋮   │   ⋮      ⋮      ⋮             ⋮             ⋮       ⋮         ⋮         ⋮         ⋮          ⋮                ⋮                        ⋮
 26673 │  2010     11      6                    8     13  BA               0         0      36.0     -110.0              -1.05336                 -0.250324
 26674 │  2013      9     14                   36     12  NA               0         0      35.5     -107.5               0.213222                 0.12
 26675 │  2023      2     18                  116     12  DS               0         0      35.5     -107.5              -0.217475                 0.042571
 26676 │  2014     11     22                   49     13  PF               0         0      36.0     -110.0               0.613491                -1.17076
                                                                                                                                            26668 rows omitted

Small Dataset

5335×12 DataFrame
  Row │ Year   Month  Day    Sampling_date_order  plot   Species  Abundance  Presence  Latitude  Longitude  standardized_temperature  standardized_precipitation 
      │ Int64  Int64  Int64  Int64                Int64  String3  Int64      Int64     Float64   Float64    Float64                 Float64                  
──────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    1 │  2015      7     18                   56     12  PM               0         0      35.5     -107.5               -1.30965                 0.967859
    2 │  2016      8      6                   66     13  SF               0         0      36.0     -110.0                1.45692                 1.77253
    3 │  2017      2     25                   71     21  SF               0         0      36.5     -109.0               -1.50086                 0.993311
    4 │  2018      5     19                   82     16  PB               0         0      36.0     -108.5               -1.2202                  0.684295
  ⋮   │   ⋮      ⋮      ⋮             ⋮             ⋮       ⋮         ⋮         ⋮         ⋮          ⋮                ⋮                        ⋮
 5332 │  2013     11      2                   37     13  RF               0         0      36.0     -110.0                2.07639                 2.72841
 5333 │  2018     11     10                   88      6  PH               0         0      35.0     -107.5               -0.197006                0.842547
 5334 │  2018      3     24                   80      2  DS               0         0      35.0     -109.5               -0.920093               -0.276074
 5335 │  2014      4     26                   42      7  PH               0         0      35.5     -110.0                0.848755               -0.247144
                                                                                                                                            5327 rows omitted

Remarks

• For DNCI_multigroup_result, 100 permutations per sample are used in both the Julia and R implementation, and parallelComputing was set to be TRUE when benchmarking DNCImper:::DNCI_multigroup() in R. This means the R implementation distributes permutations across multiple cores, reducing the peak memory footprint per core, which likely contributes to the substantially lower memory usage reported for R compared to Julia. Additionally, direct memory comparisons should be interpreted with caution as Julia reports total memory allocated during execution while R only tracks heap allocations.

The Scripts Used for Benchmarking

• Julia
• R

Packages used for benchmarking

• bench
• BenchmarkTools.jl