The shader simply does a cross product of the input colour and a fixed vector (though one that varies as you move the mouse) to generate another colour. So the colours range over almost all possible colours they are treated as vectors around a colour (0.5, 0.5, 0.5), i.e. 0x808080 or mid-grey.
The function in full, including the normalisation and cross product, is
float4 col = sampleNearest(src, outCoord());
float3 ortho = col.xyz;
ortho = ortho * 2.0 - float3(1.0, 1.0, 1.0);
ortho = cross(ortho, adj);
ortho = (ortho + float3(1.0, 1.0, 1.0)) * 0.5;
dst = float4(ortho.x, ortho.y, ortho.z, 1.0);
As moving the mouse only updates two coordinates one of the values passed to the shader is fixed. Clicking changes which it is, copying the fixed value from the last y value and changes which two colours moving the mouse affects. See the source for details; the full shader source is embedded at the end of the Flash example code.
As noted above I can't see a real use for this as it is, but the performance is good enough that it could be made more complex and still be usable. Possible improvements include changing it to rotate (so all colour information is preserved) or changing it to work in another colour space where fixing one axis will have more meaning. Or even using the colour from a second image rather than a fixed vector as input.