I have a 2017 Impreza Sport CVT 2.0 and was skeptical of the CVT at first. I had test driven an Outback CVT several years earlier and wasn't impressed with that earlier version, but they've really improved the CVT post 2017. I've driven several 4EATs ranging from 1996-2017, including a WRX (with a VTD 4EAT) and they really weren't as fun - which is surprising considering this car only has 140hp and no SI drive. I would love to try a WRX with the SPT and SI drive, I'm sure I'd like that.
I've got 125,000 miles on this car, and I've probably done 20% of that in manual mode. Normal commuting is mostly in "D". Some times I'll slip into manual to accelerate from a stop, when I don't want to deal with the car upshifting as early; around 20mph it really wants to drop the rpms down to ~1200 unless I'm heavy on the accelerator. Same on a long ascent or decent, or stop-and-go traffic, if the car is hunting gears a little too much for my liking. Taking any corners quickly is better when you preselect your gear. Sometimes merging onto the freeway, as the car has no real power.
I'm in manual the most when I'm having fun. Every Saturday morning, I'll pick a fun road or mountain that I can get to within a day. Usually I'll do 200-400 miles, around 6-12 hours depending on my mood. I'm in the California, Bay Area, so there's plenty to choose from. For this I'm only in "D" for the commute out/back and in "M" most of the time I'm somewhere twisty.
The car is much more controllable in manual on windy roads. Uphill the throttle response is more linear, downhill the speed is much more controlled in 1st and 2nd gear and lets me tap the brakes if I'm coming in a little too fast without upsetting the cars balance. 1st and 2nd gear the car is also very neutral, letting me feather the throttle for a bit of power-on oversteer (like a rear wheel drive car). Not nearly as aggressive (well, 1st gear is), but nice when I just want a little extra slip angle on slower corners (e.g. hairpins).
I've done deep snow and soft dirt only once or twice. The Impreza (and BRZ) are definitely better suited to asphalt. The dirt/gravel road definitely made me wish I had the tires and ground clearance of the Crosstrek. But I prefer pavement so the Impreza suits me better.
Honestly, I'd suggest everyone with the paddle shifters to at least practice using manual mode or semi-auto mode (paddle shift in "D") so you're not scared to use it in the situations where if it might be beneficial. Even if you do so only rarely.