Digital Out is for the optical digital out port on the back of your PS2 and you would need to buy a Fibre optic cable to hook to your Dolby Digital comaptiable Surround Sound receiver
RGB is your standard video output mode and is what you probably should set it to,