I'd push against your suggestion because you can get risk-free interest of more than 4% (e.g., SoFi savings if you have any size direct deposit), so you get half as much again return on your savings as you'd save in mortgage interest.