Short answer, yes. Everything will bolt up.
You may also need to swap the rear diff, if the cars have different final drive ratios. Use the Legacy driveshaft either way.
Gonna want the flexplate that fits the torque converter that fits the trans you're using, so all that from the RS.
Electronically, I'm not sure if the Legacy TCU will control the newer RS trans. There are differences from phase I to phase II, and phase II has 2 versions, but I forgot the years/changes. You may have to swap the RS TCU over as well.
Edit: found a thread I was recently reading, chock full of info on the 4eat. It's about swapping a 5eat into an SVX, but good info regardless.
4EAT Phase II Version II Swap - The Subaru SVX World Network