next up previous contents
Next: WP6: Conclusion Up: Task 6.3: Algorithms Porting Previous: Task 6.3: Status

Task 6.3: Technical Description

The work carried out to produce a SPERT version of RNN training is detailed below:

1.
Workstation version of the RAP code using the optimized floating-point vector/matrix library from ICSI (fltvec). [September 1996]

2.
SPERT version using the fixed-point vector/matrix library (fxlib). This version performs all arithmetic in 16-bit fixed-point. Lack of precision caused unstable training and the acoustic models to result in approximately 100% increase in word error rate. [January 1997]

3.
Fixed-point arithmetic changed to 32-bit where supported by the hardware and the fixed-point library. This failed to solve the problems seen in the 16-bit version. [February 1997]

4.
Those areas where both dynamic range and precision were considered important were changed to floating-point. This solved the stability problem, but training was very slow due to the need to do floating-point emulation (SPERT does not have hardware support for floating-point). [April 1997]

5.
A complete re-write of the RNN training software to use data structures more suitable for fixed-point arithmetic (separate data items for all values whose ranges and/or required precision are different). This was initially written using in floating-point to run on workstations. [June 1997]

6.
Converted new version into 16-bit fixed-point. Still had problems with stability during training. A 32-bit version was also written, but this did not solve the stability problem. The underlying problem with all the fixed-point versions is the need to accumulate a large number of small gradients. This requires the ability to accurately represent small values and also a large dynamic range. ICSI provided direct assistance with setting of exponents and in the use of a mix of 16 and 32-bit arithmetic. [August 1997]


next up previous contents
Next: WP6: Conclusion Up: Task 6.3: Algorithms Porting Previous: Task 6.3: Status
Christophe Ris
1998-11-10