Finally I figured out, why the rotation of the upper and lower leg bone were wrong. The solution looks for me like there is a unity3d bug in Transform.Rotate(), because I wanted to rotate around the local y-axis of the bone, but I have to use the world space.