I have owned both (I assume you are referring to the "old" L320 RRS and not the new L494 platform that came out in 2014?) There is virtually no difference between the two vehicles, as there shouldn't be since both are based on the same platform - slightly shortened for the Sport - and use the same exact on and off-road technologies. The Sport has a shorter wheelbase, which can be an asset off road (more agile and smaller turning radius) but makes the ride on pavement feel a bit harsher and somewhat "busier" than the LR3/4.
Despite the overall different looks and dimensions, the two vehicles weigh almost exactly the same. The Sport has an edge in on-road handling because it's more compact, comes with lower profile tires and has a bit sportier suspensions setup (especially the Supercharged model with extra ponies, Dynamic Response active sway bars and larger front Brembo brakes), while the LR3/4 offers a more plush and quieter ride, along with acres of extra cargo room of course, at the expense of less crisp handling. Surprisingly, passenger room in the first 2 rows is about the same, but the feeling behind the wheel is completely different between the two vehicles - you have to drive both to appreciate it. On the Sport, you are cocooned inside a cabin that makes you feel like you are in a sports car (some love it, some find it a bit tight and claustrophobic), the LR4 feels like a truck, with plenty of space around you, especially if you are a big and tall guy. A test drive is definitely a good idea.
Off road it's pretty much a tossup, if the two are equipped similarly (as hard as it is to find a LR3/4 with locking rear diff for example, it's almost imppssible to find a Sport with one... It's an option as well, but nobody wants it in that car.) The LR3/4 offers marginally better approach and departure angles, but the Sport makes up for those deficiencies with a shorter wheelbase. The LR4 is a better choice off-road only because of all the extra space, plus it enjoys a lot more aftermarket support than the Sport. But when equipped similarly, neither car offers meaningful advantages over the other off the pavement. They are both hugely capable and you will be able to go anywhere the other one goes.
The new 2014+ Sport is a different animal, if that's what you are comparing then I can't offer an objective opinion, but being a moderator on a Range Rover forum I would caution you that the new platform definitely has quite a few "teething" problems, which are fairly normal for a new platform, especially from Land Rover...