2014-04-17 13:51:03 +03:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2013-2014 Mozilla Corporation
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
* Copyright (c) 2017 Rostislav Pehlivanov <atomnuker@gmail.com>
|
2014-04-17 13:51:03 +03:00
|
|
|
*
|
2014-05-15 15:50:12 +03:00
|
|
|
* This file is part of FFmpeg.
|
2014-04-17 13:51:03 +03:00
|
|
|
*
|
2014-05-15 15:50:12 +03:00
|
|
|
* FFmpeg is free software; you can redistribute it and/or
|
2014-04-17 13:51:03 +03:00
|
|
|
* modify it under the terms of the GNU Lesser General Public
|
|
|
|
* License as published by the Free Software Foundation; either
|
|
|
|
* version 2.1 of the License, or (at your option) any later version.
|
|
|
|
*
|
2014-05-15 15:50:12 +03:00
|
|
|
* FFmpeg is distributed in the hope that it will be useful,
|
2014-04-17 13:51:03 +03:00
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
|
|
* Lesser General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU Lesser General Public
|
2014-05-15 15:50:12 +03:00
|
|
|
* License along with FFmpeg; if not, write to the Free Software
|
2014-04-17 13:51:03 +03:00
|
|
|
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @file
|
|
|
|
* Celt non-power of 2 iMDCT
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <float.h>
|
|
|
|
#include <math.h>
|
2014-04-28 18:56:43 +03:00
|
|
|
#include <stddef.h>
|
2021-08-01 08:36:09 +02:00
|
|
|
#include <stdint.h>
|
2014-04-28 18:56:43 +03:00
|
|
|
|
|
|
|
#include "config.h"
|
2014-04-17 13:51:03 +03:00
|
|
|
|
|
|
|
#include "libavutil/attributes.h"
|
2021-08-01 08:36:09 +02:00
|
|
|
#include "libavutil/error.h"
|
2014-04-17 13:51:03 +03:00
|
|
|
|
2017-02-01 05:13:06 +02:00
|
|
|
#include "mdct15.h"
|
2014-04-17 13:51:03 +03:00
|
|
|
|
2017-04-01 20:41:52 +02:00
|
|
|
#define FFT_FLOAT 1
|
|
|
|
#include "fft-internal.h"
|
2014-04-17 13:51:03 +03:00
|
|
|
|
2017-04-01 20:41:52 +02:00
|
|
|
#define CMUL3(c, a, b) CMUL((c).re, (c).im, (a).re, (a).im, (b).re, (b).im)
|
2014-04-17 13:51:03 +03:00
|
|
|
|
2017-02-01 05:13:06 +02:00
|
|
|
av_cold void ff_mdct15_uninit(MDCT15Context **ps)
|
2014-04-17 13:51:03 +03:00
|
|
|
{
|
2017-02-01 05:13:06 +02:00
|
|
|
MDCT15Context *s = *ps;
|
2014-04-17 13:51:03 +03:00
|
|
|
|
|
|
|
if (!s)
|
|
|
|
return;
|
|
|
|
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
ff_fft_end(&s->ptwo_fft);
|
2014-04-17 13:51:03 +03:00
|
|
|
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
av_freep(&s->pfa_prereindex);
|
|
|
|
av_freep(&s->pfa_postreindex);
|
2014-04-17 13:51:03 +03:00
|
|
|
av_freep(&s->twiddle_exptab);
|
|
|
|
av_freep(&s->tmp);
|
|
|
|
|
|
|
|
av_freep(ps);
|
|
|
|
}
|
|
|
|
|
2017-02-01 05:13:06 +02:00
|
|
|
static inline int init_pfa_reindex_tabs(MDCT15Context *s)
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
{
|
|
|
|
int i, j;
|
|
|
|
const int b_ptwo = s->ptwo_fft.nbits; /* Bits for the power of two FFTs */
|
|
|
|
const int l_ptwo = 1 << b_ptwo; /* Total length for the power of two FFTs */
|
|
|
|
const int inv_1 = l_ptwo << ((4 - b_ptwo) & 3); /* (2^b_ptwo)^-1 mod 15 */
|
|
|
|
const int inv_2 = 0xeeeeeeef & ((1U << b_ptwo) - 1); /* 15^-1 mod 2^b_ptwo */
|
|
|
|
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
s->pfa_prereindex = av_malloc_array(15 * l_ptwo, sizeof(*s->pfa_prereindex));
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
if (!s->pfa_prereindex)
|
|
|
|
return 1;
|
|
|
|
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
s->pfa_postreindex = av_malloc_array(15 * l_ptwo, sizeof(*s->pfa_postreindex));
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
if (!s->pfa_postreindex)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
/* Pre/Post-reindex */
|
|
|
|
for (i = 0; i < l_ptwo; i++) {
|
|
|
|
for (j = 0; j < 15; j++) {
|
|
|
|
const int q_pre = ((l_ptwo * j)/15 + i) >> b_ptwo;
|
|
|
|
const int q_post = (((j*inv_1)/15) + (i*inv_2)) >> b_ptwo;
|
2017-03-22 19:17:04 +02:00
|
|
|
const int k_pre = 15*i + (j - q_pre*15)*(1 << b_ptwo);
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
const int k_post = i*inv_2*15 + j*inv_1 - 15*q_post*l_ptwo;
|
2017-07-25 05:31:48 +02:00
|
|
|
s->pfa_prereindex[i*15 + j] = k_pre << 1;
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
s->pfa_postreindex[k_post] = l_ptwo*j + i;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Stride is hardcoded to 3 */
|
2017-06-18 13:06:30 +02:00
|
|
|
static inline void fft5(FFTComplex *out, FFTComplex *in, FFTComplex exptab[2])
|
2014-04-17 13:51:03 +03:00
|
|
|
{
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
FFTComplex z0[4], t[6];
|
|
|
|
|
|
|
|
t[0].re = in[3].re + in[12].re;
|
|
|
|
t[0].im = in[3].im + in[12].im;
|
|
|
|
t[1].im = in[3].re - in[12].re;
|
|
|
|
t[1].re = in[3].im - in[12].im;
|
|
|
|
t[2].re = in[6].re + in[ 9].re;
|
|
|
|
t[2].im = in[6].im + in[ 9].im;
|
|
|
|
t[3].im = in[6].re - in[ 9].re;
|
|
|
|
t[3].re = in[6].im - in[ 9].im;
|
|
|
|
|
|
|
|
out[0].re = in[0].re + in[3].re + in[6].re + in[9].re + in[12].re;
|
|
|
|
out[0].im = in[0].im + in[3].im + in[6].im + in[9].im + in[12].im;
|
|
|
|
|
|
|
|
t[4].re = exptab[0].re * t[2].re - exptab[1].re * t[0].re;
|
|
|
|
t[4].im = exptab[0].re * t[2].im - exptab[1].re * t[0].im;
|
|
|
|
t[0].re = exptab[0].re * t[0].re - exptab[1].re * t[2].re;
|
|
|
|
t[0].im = exptab[0].re * t[0].im - exptab[1].re * t[2].im;
|
|
|
|
t[5].re = exptab[0].im * t[3].re - exptab[1].im * t[1].re;
|
|
|
|
t[5].im = exptab[0].im * t[3].im - exptab[1].im * t[1].im;
|
|
|
|
t[1].re = exptab[0].im * t[1].re + exptab[1].im * t[3].re;
|
|
|
|
t[1].im = exptab[0].im * t[1].im + exptab[1].im * t[3].im;
|
|
|
|
|
|
|
|
z0[0].re = t[0].re - t[1].re;
|
|
|
|
z0[0].im = t[0].im - t[1].im;
|
|
|
|
z0[1].re = t[4].re + t[5].re;
|
|
|
|
z0[1].im = t[4].im + t[5].im;
|
|
|
|
|
|
|
|
z0[2].re = t[4].re - t[5].re;
|
|
|
|
z0[2].im = t[4].im - t[5].im;
|
|
|
|
z0[3].re = t[0].re + t[1].re;
|
|
|
|
z0[3].im = t[0].im + t[1].im;
|
|
|
|
|
|
|
|
out[1].re = in[0].re + z0[3].re;
|
|
|
|
out[1].im = in[0].im + z0[0].im;
|
|
|
|
out[2].re = in[0].re + z0[2].re;
|
|
|
|
out[2].im = in[0].im + z0[1].im;
|
|
|
|
out[3].re = in[0].re + z0[1].re;
|
|
|
|
out[3].im = in[0].im + z0[2].im;
|
|
|
|
out[4].re = in[0].re + z0[0].re;
|
|
|
|
out[4].im = in[0].im + z0[3].im;
|
2014-04-17 13:51:03 +03:00
|
|
|
}
|
|
|
|
|
2017-06-18 13:06:30 +02:00
|
|
|
static void fft15_c(FFTComplex *out, FFTComplex *in, FFTComplex *exptab, ptrdiff_t stride)
|
2014-04-17 13:51:03 +03:00
|
|
|
{
|
|
|
|
int k;
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
FFTComplex tmp1[5], tmp2[5], tmp3[5];
|
2014-04-17 13:51:03 +03:00
|
|
|
|
2017-06-18 13:06:30 +02:00
|
|
|
fft5(tmp1, in + 0, exptab + 19);
|
|
|
|
fft5(tmp2, in + 1, exptab + 19);
|
|
|
|
fft5(tmp3, in + 2, exptab + 19);
|
2014-04-17 13:51:03 +03:00
|
|
|
|
|
|
|
for (k = 0; k < 5; k++) {
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
FFTComplex t[2];
|
|
|
|
|
2017-04-01 20:41:52 +02:00
|
|
|
CMUL3(t[0], tmp2[k], exptab[k]);
|
|
|
|
CMUL3(t[1], tmp3[k], exptab[2 * k]);
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
out[stride*k].re = tmp1[k].re + t[0].re + t[1].re;
|
|
|
|
out[stride*k].im = tmp1[k].im + t[0].im + t[1].im;
|
|
|
|
|
2017-04-01 20:41:52 +02:00
|
|
|
CMUL3(t[0], tmp2[k], exptab[k + 5]);
|
|
|
|
CMUL3(t[1], tmp3[k], exptab[2 * (k + 5)]);
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
out[stride*(k + 5)].re = tmp1[k].re + t[0].re + t[1].re;
|
|
|
|
out[stride*(k + 5)].im = tmp1[k].im + t[0].im + t[1].im;
|
|
|
|
|
2017-04-01 20:41:52 +02:00
|
|
|
CMUL3(t[0], tmp2[k], exptab[k + 10]);
|
|
|
|
CMUL3(t[1], tmp3[k], exptab[2 * k + 5]);
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
out[stride*(k + 10)].re = tmp1[k].re + t[0].re + t[1].re;
|
|
|
|
out[stride*(k + 10)].im = tmp1[k].im + t[0].im + t[1].im;
|
2014-04-17 13:51:03 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-02-01 05:13:06 +02:00
|
|
|
static void mdct15(MDCT15Context *s, float *dst, const float *src, ptrdiff_t stride)
|
|
|
|
{
|
|
|
|
int i, j;
|
|
|
|
const int len4 = s->len4, len3 = len4 * 3, len8 = len4 >> 1;
|
|
|
|
const int l_ptwo = 1 << s->ptwo_fft.nbits;
|
|
|
|
FFTComplex fft15in[15];
|
|
|
|
|
|
|
|
/* Folding and pre-reindexing */
|
|
|
|
for (i = 0; i < l_ptwo; i++) {
|
|
|
|
for (j = 0; j < 15; j++) {
|
|
|
|
const int k = s->pfa_prereindex[i*15 + j];
|
2017-07-25 05:31:48 +02:00
|
|
|
FFTComplex tmp, exp = s->twiddle_exptab[k >> 1];
|
|
|
|
if (k < len4) {
|
|
|
|
tmp.re = -src[ len4 + k] + src[1*len4 - 1 - k];
|
|
|
|
tmp.im = -src[ len3 + k] - src[1*len3 - 1 - k];
|
2017-02-01 05:13:06 +02:00
|
|
|
} else {
|
2017-07-25 05:31:48 +02:00
|
|
|
tmp.re = -src[ len4 + k] - src[5*len4 - 1 - k];
|
|
|
|
tmp.im = src[-len4 + k] - src[1*len3 - 1 - k];
|
2017-02-01 05:13:06 +02:00
|
|
|
}
|
2017-07-25 05:31:48 +02:00
|
|
|
CMUL(fft15in[j].im, fft15in[j].re, tmp.re, tmp.im, exp.re, exp.im);
|
2017-02-01 05:13:06 +02:00
|
|
|
}
|
2017-06-18 13:06:30 +02:00
|
|
|
s->fft15(s->tmp + s->ptwo_fft.revtab[i], fft15in, s->exptab, l_ptwo);
|
2017-02-01 05:13:06 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Then a 15xN FFT (where N is a power of two) */
|
|
|
|
for (i = 0; i < 15; i++)
|
|
|
|
s->ptwo_fft.fft_calc(&s->ptwo_fft, s->tmp + l_ptwo*i);
|
|
|
|
|
|
|
|
/* Reindex again, apply twiddles and output */
|
|
|
|
for (i = 0; i < len8; i++) {
|
|
|
|
const int i0 = len8 + i, i1 = len8 - i - 1;
|
|
|
|
const int s0 = s->pfa_postreindex[i0], s1 = s->pfa_postreindex[i1];
|
|
|
|
|
2017-07-25 05:31:48 +02:00
|
|
|
CMUL(dst[2*i1*stride + stride], dst[2*i0*stride], s->tmp[s0].re, s->tmp[s0].im,
|
|
|
|
s->twiddle_exptab[i0].im, s->twiddle_exptab[i0].re);
|
|
|
|
CMUL(dst[2*i0*stride + stride], dst[2*i1*stride], s->tmp[s1].re, s->tmp[s1].im,
|
|
|
|
s->twiddle_exptab[i1].im, s->twiddle_exptab[i1].re);
|
2017-02-01 05:13:06 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void imdct15_half(MDCT15Context *s, float *dst, const float *src,
|
2017-07-11 22:29:22 +02:00
|
|
|
ptrdiff_t stride)
|
2014-04-17 13:51:03 +03:00
|
|
|
{
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
FFTComplex fft15in[15];
|
2014-04-17 13:51:03 +03:00
|
|
|
FFTComplex *z = (FFTComplex *)dst;
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
int i, j, len8 = s->len4 >> 1, l_ptwo = 1 << s->ptwo_fft.nbits;
|
|
|
|
const float *in1 = src, *in2 = src + (s->len2 - 1) * stride;
|
|
|
|
|
|
|
|
/* Reindex input, putting it into a buffer and doing an Nx15 FFT */
|
|
|
|
for (i = 0; i < l_ptwo; i++) {
|
|
|
|
for (j = 0; j < 15; j++) {
|
|
|
|
const int k = s->pfa_prereindex[i*15 + j];
|
2017-07-25 05:31:48 +02:00
|
|
|
FFTComplex tmp = { in2[-k*stride], in1[k*stride] };
|
|
|
|
CMUL3(fft15in[j], tmp, s->twiddle_exptab[k >> 1]);
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
}
|
2017-06-18 13:06:30 +02:00
|
|
|
s->fft15(s->tmp + s->ptwo_fft.revtab[i], fft15in, s->exptab, l_ptwo);
|
2014-04-17 13:51:03 +03:00
|
|
|
}
|
|
|
|
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
/* Then a 15xN FFT (where N is a power of two) */
|
|
|
|
for (i = 0; i < 15; i++)
|
|
|
|
s->ptwo_fft.fft_calc(&s->ptwo_fft, s->tmp + l_ptwo*i);
|
2014-04-17 13:51:03 +03:00
|
|
|
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
/* Reindex again, apply twiddles and output */
|
|
|
|
s->postreindex(z, s->tmp, s->twiddle_exptab, s->pfa_postreindex, len8);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void postrotate_c(FFTComplex *out, FFTComplex *in, FFTComplex *exp,
|
|
|
|
int *lut, ptrdiff_t len8)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
/* Reindex again, apply twiddles and output */
|
2014-04-17 13:51:03 +03:00
|
|
|
for (i = 0; i < len8; i++) {
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
const int i0 = len8 + i, i1 = len8 - i - 1;
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
const int s0 = lut[i0], s1 = lut[i1];
|
imdct15: replace the FFT with a faster PFA FFT algorithm
This commit replaces the current inefficient non-power-of-two FFT with a
much faster FFT based on the Prime Factor Algorithm.
Although it is already much faster than the old algorithm without SIMD,
the new algorithm makes use of the already very throughouly SIMD'd power
of two FFT, which improves performance even more across all platforms
which we have SIMD support for.
Most of the work was done by Peter Barfuss, who passed the code to me to
implement into the iMDCT and the current codebase. The code for a
5-point and 15-point FFT was derived from the previous implementation,
although it was optimized and simplified, which will make its future
SIMD easier. The 15-point FFT is currently using 6% of the current
overall decoder overhead.
The FFT can now easily be used as a forward transform by simply not
multiplying the 5-point FFT's imaginary component by -1 (which comes
from the fact that changing the complex exponential's angle by -1 also
changes the output by that) and by multiplying the "theta" angle of the
main exptab by -1. Hence the deliberately left multiplication by -1 at
the end.
FATE passes, and performance reports on other platforms/CPUs are
welcome.
Performance comparisons:
iMDCT, PFA:
101127 decicycles in speed, 32765 runs, 3 skips
iMDCT, Old:
211022 decicycles in speed, 32768 runs, 0 skips
Standalone FFT, 300000 transforms of size 960:
PFA Old FFT kiss_fft libfftw3f
3.659695s, 15.726912s, 13.300789s, 1.182222s
Being only 3x slower than libfftw3f is a big achievement by itself.
There appears to be something capping the performance in the iMDCT side
of things, possibly during the pre-stage reindexing. However, it is
certainly fast enough for now.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-01-04 11:23:24 +02:00
|
|
|
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
CMUL(out[i1].re, out[i0].im, in[s1].im, in[s1].re, exp[i1].im, exp[i1].re);
|
|
|
|
CMUL(out[i0].re, out[i1].im, in[s0].im, in[s0].re, exp[i0].im, exp[i0].re);
|
2014-04-17 13:51:03 +03:00
|
|
|
}
|
|
|
|
}
|
2017-06-18 13:06:30 +02:00
|
|
|
|
|
|
|
av_cold int ff_mdct15_init(MDCT15Context **ps, int inverse, int N, double scale)
|
|
|
|
{
|
|
|
|
MDCT15Context *s;
|
|
|
|
double alpha, theta;
|
|
|
|
int len2 = 15 * (1 << N);
|
|
|
|
int len = 2 * len2;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Tested and verified to work on everything in between */
|
|
|
|
if ((N < 2) || (N > 13))
|
|
|
|
return AVERROR(EINVAL);
|
|
|
|
|
|
|
|
s = av_mallocz(sizeof(*s));
|
|
|
|
if (!s)
|
|
|
|
return AVERROR(ENOMEM);
|
|
|
|
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
s->fft_n = N - 1;
|
|
|
|
s->len4 = len2 / 2;
|
|
|
|
s->len2 = len2;
|
|
|
|
s->inverse = inverse;
|
|
|
|
s->fft15 = fft15_c;
|
|
|
|
s->mdct = mdct15;
|
|
|
|
s->imdct_half = imdct15_half;
|
|
|
|
s->postreindex = postrotate_c;
|
2017-06-18 13:06:30 +02:00
|
|
|
|
|
|
|
if (ff_fft_init(&s->ptwo_fft, N - 1, s->inverse) < 0)
|
|
|
|
goto fail;
|
|
|
|
|
|
|
|
if (init_pfa_reindex_tabs(s))
|
|
|
|
goto fail;
|
|
|
|
|
|
|
|
s->tmp = av_malloc_array(len, 2 * sizeof(*s->tmp));
|
|
|
|
if (!s->tmp)
|
|
|
|
goto fail;
|
|
|
|
|
mdct15: add inverse transform postrotation SIMD
2.5ms frames:
Before (c): 2638 decicycles in postrotate, 2097040 runs, 112 skips
After (sse3): 1467 decicycles in postrotate, 2097083 runs, 69 skips
After (avx2): 1244 decicycles in postrotate, 2097085 runs, 67 skips
5ms frames:
Before (c): 4987 decicycles in postrotate, 1048371 runs, 205 skips
After (sse3): 2644 decicycles in postrotate, 1048509 runs, 67 skips
After (avx2): 2031 decicycles in postrotate, 1048523 runs, 53 skips
10ms frames:
Before (c): 9153 decicycles in postrotate, 523575 runs, 713 skips
After (sse3): 5110 decicycles in postrotate, 523726 runs, 562 skips
After (avx2): 3738 decicycles in postrotate, 524223 runs, 65 skips
20ms frames:
Before (c): 17857 decicycles in postrotate, 261866 runs, 278 skips
After (sse3): 10041 decicycles in postrotate, 261746 runs, 398 skips
After (avx2): 7050 decicycles in postrotate, 262116 runs, 28 skips
Improves total decoding performance for real world content by 9% with avx2.
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2017-07-29 22:27:01 +02:00
|
|
|
s->twiddle_exptab = av_malloc_array(s->len4, sizeof(*s->twiddle_exptab));
|
2017-06-18 13:06:30 +02:00
|
|
|
if (!s->twiddle_exptab)
|
|
|
|
goto fail;
|
|
|
|
|
|
|
|
theta = 0.125f + (scale < 0 ? s->len4 : 0);
|
|
|
|
scale = sqrt(fabs(scale));
|
|
|
|
for (i = 0; i < s->len4; i++) {
|
|
|
|
alpha = 2 * M_PI * (i + theta) / len;
|
|
|
|
s->twiddle_exptab[i].re = cosf(alpha) * scale;
|
|
|
|
s->twiddle_exptab[i].im = sinf(alpha) * scale;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* 15-point FFT exptab */
|
|
|
|
for (i = 0; i < 19; i++) {
|
|
|
|
if (i < 15) {
|
|
|
|
double theta = (2.0f * M_PI * i) / 15.0f;
|
|
|
|
if (!s->inverse)
|
|
|
|
theta *= -1;
|
|
|
|
s->exptab[i].re = cosf(theta);
|
|
|
|
s->exptab[i].im = sinf(theta);
|
|
|
|
} else { /* Wrap around to simplify fft15 */
|
|
|
|
s->exptab[i] = s->exptab[i - 15];
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* 5-point FFT exptab */
|
|
|
|
s->exptab[19].re = cosf(2.0f * M_PI / 5.0f);
|
|
|
|
s->exptab[19].im = sinf(2.0f * M_PI / 5.0f);
|
|
|
|
s->exptab[20].re = cosf(1.0f * M_PI / 5.0f);
|
|
|
|
s->exptab[20].im = sinf(1.0f * M_PI / 5.0f);
|
|
|
|
|
|
|
|
/* Invert the phase for an inverse transform, do nothing for a forward transform */
|
|
|
|
if (s->inverse) {
|
|
|
|
s->exptab[19].im *= -1;
|
|
|
|
s->exptab[20].im *= -1;
|
|
|
|
}
|
|
|
|
|
2022-06-12 05:51:12 +02:00
|
|
|
#if ARCH_X86
|
|
|
|
ff_mdct15_init_x86(s);
|
|
|
|
#endif
|
2017-06-18 13:06:30 +02:00
|
|
|
|
|
|
|
*ps = s;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
fail:
|
|
|
|
ff_mdct15_uninit(&s);
|
|
|
|
return AVERROR(ENOMEM);
|
|
|
|
}
|