Cats generally have little trouble finding and using a litter box, but sometimes cats might defecate or urinate outside of the litter box. When that happens, it could be a variety of problems beyond just your cat being mad at you.
First, try using different liter since cats might be picky about your current litter choice. Second, make sure you clean the litter box regularly since cats dislike dirty litter boxes that stink. (How many people like using a dirty restroom in a smelly gas station?)