I know that my mom's been eating nearly entirely dairy free for about a year and she's enjoying the benefits of it. There's a substitution for lactose food and drink that allows a similar diet as you may currently have, yet without the dairy. And real talk, try out almond milk. That's that good stuff.
Also, if you're talking putting butter on wheat products, use Nutella instead. Heck, put Nutella on everything.