FSMP 在 JUNO commisson 的显存超出现象

/cc @liuja

在 JUNO commissioning 分析中发现的例子。

cd /mnt/stage/JUNO/commisson-0104/FSMP
./fsmp.py sparsify/2280/074.h5 -o fsmp/2280/074.pq --ref SER.h5 --full PE/2280/135.pq
Analyze waveforms, 144431 in total.
Part 0 | 0:135936 with lengths of PE t 45, waveform 122 and NPE 18
Starting batch loop...
Summarising...
Part 1 | 135936:142717 with lengths of PE t 414, waveform 491 and NPE 29
Starting batch loop...
Summarising...
Part 2 | 142717:144368 with lengths of PE t 901, waveform 977 and NPE 39
Starting batch loop...

  File "/mnt/stage/JUNO/commisson-0104/FSMP/analysis/fsmp/batch.py", line 143, in batch
    F -= cp.transpose(ATUTU, (0, 2, 1)) @ cp.linalg.solve(AUUA, ATUTU)
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 5,384,961,536 bytes (allocated so far: 28,885,370,368 bytes).

同样的问题还有 fsmp/2280/082.pq。

估算显存时，应该是没有考虑 full output 情形，没有把 MMSE 所占显存也计算进来。