The engine keeps three registers. x and y hold a vector, and z holds the angle still left to turn. On stage i the vector turns by plus or minus atan(2^-i). A turn of that size equals x shifted right by i bits, added to or taken from y, and the other way round for x. A small lookup table holds the eight angles.
In rotation mode the sign of z picks the turn direction, and z shrinks toward zero. Starting from x = 0.607 cancels the length growth of 1.6467 that the eight turns add, so the result is cosine in x and sine in y.
In vectoring mode the sign of y picks the direction and the engine drives y to zero. Then z holds the angle of the input and x holds its length times 1.6467. A vector in the left half plane is first turned by 90 degrees so the loop converges.