There are three important things here, in my opinion.

1. That you understand the risk you take on by counting on your investments to serve as your emergency fund.

2. That the level of risk is appropriate given your financial situation.

3. That you make an informed choice to accept that risk.

Once these three things are taken care of, the opinions of others (except for your spouse ;)) don't matter.