It's my understanding that if you own the property, and have the right blueprints anything is possible. I doubt the Counts and barons will be too keen on letting random players build their house too big, or too ugly and that may certainly be a major constraint on what can be done in any given area.
As for building on someone else's property, well if you have their permission I don't see why not. But it's been said that construction is a multi-person project and it will take time, so don't think you can do it behind the property owners back.