Visuals matter, even at the playtesting stage. First and foremost, they attract testers. People will stop and ask about a visually appealing game, whereas you’ll have to work to drum people up for a game that’s scrawled on a sheet of paper.
Second, they help teach the rules. If all of your components are 3″ x 5″ cards with numbers on them, your game will seem more complicated than it is. Using some graphic design will help testers engage with the game in a deeper way, since they won’t be wasting energy trying to remember which featureless cards go on the left and which go on the right.
Don’t waste time making art for every prototype, especially early ones. When you have an idea that’s working, though, take the time to do some simple graphics for it. Your process will benefit a great deal.