There is no single answer, it depends on your PC specs, just mess with the bitrate and stuff until you get an acceptable result, or upgrade your PC
Game setting: Any that your PC can handle. The higher, the better of course. Recorder setting: High bitrate (60mbps and up). Will lead to higher file size, but also higher quality, especially with detail-dense clips. For reference, I use Nvidia Shadowplay mostly with it set at 120mbps.