When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.
Unfortunately you definitely have to remove the door panel. The speaker grill has some little metal tabs that are folded behind the door panel and that is the only way to get to them.
The door panel is pretty easy to take off, just read these instructions.
The panel does have to come off. It is held by some screws around the trim pieces and plastic connectors that are basically a spike on the panel and a receiver on the door. I would recommend picking up some replacement ones before you take the panel off; you'll break at least a couple, guaranteed (they're pretty flimsy).
with what they said. It is actually pretty easy to get the doors off on our cars. On mine they were a large size (T40-50?) torx under the little plastic cover behind the door handle, remove those and the rest of the door just pops off.
Removing the speaker cover after that is easy too, just have to bend the metal tabs holding it on!