Which is more important to the story "The Yellow Wallpaper"—the character development or the setting?

I think you are on the right track and I agree with your thinking so far. I would go on to argue that the setting of the story is the driving force of the protagonist's character change and descent into madness. 

First the story has many conventions that mirror the Gothic literature popular in the 1800s, not in the least the setting. The narrator describes the house as an "ancestral hall" and a "hereditary estate" in the very first paragraph – she even wants to call it a haunted house, the epitome of the Gothic tradition. Images of abandonment and ruin litter the grounds: broken greenhouses and abandoned cottages. While Gilman states that her intention was for the novel to "keep people from being driven crazy" by rest cures like the one she suffered, the narrator's insanity is a critical feature of the book, and one that follows in the tradition of Gothic novels. The Gothic setting and narrator's descent into madness are absolutely linked by genre.

Additionally, the isolation inherent in the setting plays a major role in the narrator's shift from the beginning of the novel to the end. The house is described as being:

"quite alone, standing well back from the road, quite three miles from the village. [...] there are hedges and walls and gates that lock, and lots of separate little houses for the gardeners and people"

Not only is the house itself set apart from the rest of the village, but even within the grounds and house there are all sorts of devices to keep people apart – locking gates and walls. Most isolating of all is the narrator's bedroom. Instead of the downstairs room that "opened onto the piazza," John insists that the narrator take the attic bedroom, which is so isolated that it even has bars on the windows. 

The narrator's emotional isolation begins to mirror her physical isolation. She often notes that she cannot be open and honest about her condition with John, and he frequently demonstrates his patronizing responses to her concerns. She hides her writing, stops telling him how the room affects her, and even censors her own self in her diary. By the time she is locking John and Jenny out of the room and ripping the woman out of the wallpaper, it's far too late. The isolation has driven her mad. 

Finally, the narrator's room suggests how the setting reflects her character. The room used to be a nursery, which is fitting for the way the narrator is infantilized by her husband. John calls her "blessed little goose," playfully mocks her feelings and concerns, and disregards all that she says. After the reader has become very suspicious about her mental health, the narrator asks John if they can leave the house, fearing that it is harming her. He responds that he would do so in a second if she were in any danger, but continues to say that she is getting better,

"...whether you can see it or not. I am a doctor, dear, and I know. You are gaining flesh and color, your appetite is better, I feel really much easier about you."

"I don't weigh a bit more," said I, "nor as much; and my appetite may be better in the evening when you are here, but it is worse in the morning when you are awav!"

" Bless her little heart!" said he with a big hug, "she shall be as sick as she pleases!"

They do not, clearly, leave the house. This treatment of women as foolish and childish was not uncommon in the 19th century, but it is definitely shown to be destructive here. Instead of listening to his wife's concerns, John coops her up in a former nursery, with bars on the windows – showing her real position as both child and prisoner to her husband. The "rings and things" bolted to the wall increase the sense that the room is a prison, as does the pattern on the wallpaper. Though the wallpaper at first looks random, it eventually appears to the narrator as women trapped behind the bars of the front pattern, shaking it to try to get out. As the narrator descends into madness, she moves between wanting to help the women and wanting to keep them trapped. Sometimes she even thinks she is one of them, escaped from the paper. Clearly, her isolation and husband's disdain have forced the narrator to focus on the only thing available to her, until it has driven her mad.