off course it is a big deal. Just dont do it. The more smoke and stuff built in your smoker the better tasting your next smokes will be. Most people here would only clean the racks, the drip tray, the bowl and the tray under it. Nothing else on the inside.
It is also recommended that every so often, you clean the seal on the door and the surface it touches on the tower with alcohol. I clean one more item and the would be the vent on top of the tower. You just unscrew it and clean it and the surface again with alcohol.