AVG across 3 epochs · 3 loaders × 3 datasets × 3 batch sizes
| Metric | wds-32 | wds-64 | wds-128 | str-32 | str-64 | str-128 | ray-32 | ray-64 | ray-128 | ray-pms-128 |
|---|---|---|---|---|---|---|---|---|---|---|
| samples/sec | 1,649.6 | 1,628.1 | 1,683.5 | 1,536.4 | 1,588.6 | 1,637.4 | 450.0 | 446.6 | 444.0 | 1,678.1 |
| GPU util % | 87.0 | 88.1 | 83.1 | 79.1 | 83.9 | 85.3 | 25.4 | 24.4 | 22.5 | 82.4 |
| GPU mem GB | 2.82 | 3.11 | 3.75 | 2.82 | 3.11 | 3.75 | 2.82 | 3.11 | 3.75 | 2.13 |
| CPU util % | 96.0 | 96.8 | 94.7 | 138.2 | 132.6 | 133.6 | 69.4 | 69.2 | 65.8 | 111.2 |
| RAM used GB | 1.02 | 1.03 | 1.03 | 0.97 | 0.98 | 0.98 | 3.15 | 3.23 | 3.36 | 4.13 |
| Data stall % | 27.8 | 27.3 | 28.4 | 20.5 | 27.7 | 35.5 | 72.5 | 72.5 | 73.8 | 1.32 |
| Elapsed s | 8.0 | 8.1 | 7.8 | 8.6 | 8.3 | 8.1 | 29.4 | 29.6 | 29.8 | 7.87 |
w=2 workers · bs=32/64/128 · avg across 3 epochs · 100k images · parallel decode in DataLoader workers vs Ray object store
| Metric | wds-32 | wds-64 | wds-128 | str-32 | str-64 | str-128 | ray-32 | ray-64 | ray-128 |
|---|---|---|---|---|---|---|---|---|---|
| samples/sec | 631.62 | 726.42 | 922.12 | 937.21 | 1018.54 | 1084.05 | 793.29 | 896.38 | 1136.59 |
| GPU util % | 20.30 | 21.95 | 25.62 | 30.42 | 31.67 | 32.44 | 23.45 | 26.07 | 34.82 |
| GPU mem GB | 0.950 | 1.174 | 1.623 | 0.950 | 1.174 | 1.623 | 0.950 | 1.174 | 1.623 |
| CPU util % | 1677.02 | 1259.67 | 1035.64 | 116.93 | 115.51 | 112.60 | 1631.70 | 1502.58 | 1283.17 |
| RAM used GB | 1.082 | 1.137 | 1.200 | 1.143 | 1.203 | 1.370 | 2.293 | 2.374 | 2.642 |
| Data stall % | 52.61 | 51.50 | 55.74 | 1.45 | 0.81 | 0.64 | 14.12 | 21.42 | 31.10 |
| Elapsed s | 158.33 | 137.78 | 108.45 | 106.73 | 98.17 | 92.34 | 126.25 | 111.70 | 88.03 |
| /dev/shm avg GB | 0.010 | 0.020 | 0.040 | 0.002 | 0.004 | 0.008 | 1.030 | 1.020 | 1.100 |
| /dev/shm peak GB | 0.019 | 0.039 | 0.077 | 0.010 | 0.010 | 0.019 | 1.163 | 1.163 | 1.297 |
| Metric | wds-32 | wds-64 | wds-128 | str-32 | str-64 | str-128 | ray-32 | ray-64 | ray-128 |
|---|---|---|---|---|---|---|---|---|---|
| samples/sec | 22.99 | 31.01 | 37.38 | 22.60 | 31.26 | 38.30 | 22.70 | 31.18 | 37.47 |
| GPU util % | 3.6 | 3.12 | 4.62 | 3.4 | 3.58 | 4.56 | 3.56 | 3.58 | 4.46 |
| GPU mem GB | 3.09 | 3.22 | 3.49 | 3.09 | 3.22 | 3.49 | 3.09 | 3.22 | 3.50 |
| CPU util % | 674.4 | 908.0 | 1104.1 | 670.4 | 921.8 | 1101.1 | 616.3 | 834.6 | 998.4 |
| RAM used GB | 7.29 | 4.37 | 2.98 | 7.31 | 4.34 | 2.88 | 9.39 | 6.48 | 5.00 |
| Data stall % | 0.65 | 0.86 | 1.04 | 0.44 | 0.73 | 0.91 | 3.71 | 5.20 | 6.43 |
| Elapsed s | 434.9 | 322.7 | 267.5 | 444.7 | 321.4 | 260.7 | 440.6 | 320.8 | 266.8 |