I've got 12 acres, mostly wooded three acres mowed. Kubota BX200 (cab, mower, snow thrower), and a Ford 1715 (27 old school hp), manual, 4wd, bucket and backhoe. Had them both for about 20 years now.
In my opinion, there isn't a machine that will fit both of your needs adequately. Any mower that is small, agile and economical enough for mowing is not going to have the size and power necessary for woods work, stump digging, gravel moving, carrying logs in the bucked etc.
I would try very hard to make sure any tractor you buy doesn't have an ECU and all the EPA gadgets on it .... they are a nightmare in most cases. But it will limit the size (hp rating) you can get. If you opt to get a larger machine for the woods work, shoot for something as simple and old school as possible, they are cheaper to maintain, to fix and easier to work on.
My Kubota is hydro, my Ford is manual .... in general you'll want hydro for mowing, yard work, snow throwing, garden work etc. Manual is not as big a deal with a larger tractor that you won't "driving" it nearly as much. On a larger tractor I prefer the feel you get with a clutch and selecting the gear you want to use.
I know you're a handy guy ... consider a used compact tractor (machines like the BX2200 and your Deer are considered "subcompact") Something from the 90's, pre-epa, simple and reliable .. might need some initial work (mine needed a clutch), tires etc. but you can save a ton of money .... and you won't be taking your brandy new tractor into the woods and tearing it up ... and they will get banged up doing that kind of work.
I waited till at least 10 years after we bought this land to get the larger tractor ... should have done it the first year. The Ford has paid for itself about eight times over now doing construction, digging, stump removal, logging .. I can't imagine how much we would have spent in rentals or paying people to do this kind of work (bear in mind it has a backhoe).
For doing brush we recently got a PTO driven chipper ... works great on the smaller Kubota so you can get in close to the work, or even take it down the trails to chip in the woods. Also fits the larger tractor. If you are doing tops a chipper will cut your work time by at least 50%
Here is a wall we built with the larger tractor a couple years ago .. it did the excavation, the hauling and we used it to set the 475 lb blocks ... blocks were about $37 each ... this would have cost $20K to have someone build it ... tractor paid for itself AGAIN doing just this job.
So ... go as big as you feasibly can if you end up getting a large tractor.
Tractorbynet.com is THE source for info, advice etc.