Is there any correctness rule for using "instead of" vs. "rather than" in a sentence correction problem? Or when should each be used correctly? Thanks!

'rather than' is less restrictive than 'instead of': 'rather than' can be used with just about any part of speech. so, for instance, i can write any of the following:
i chose to forsake my chance to attend law school rather than accrue major debt to enter a career about which i was ambivalent (used with verb)
[i]james differs from many negotiators in that his default attitude toward negotiation is cooperative rather than confrontational (used with adjective)
[i]i chose to rent a compact car rather than a midsize sedan (used with nouns)

'instead of', by contrast, can only be used with nouns, or with constructions that function as nouns (like gerunds and noun phrases).

--

for sentences in which either of the two constructions is acceptable, our rather extensive research indicates that the gmat prefers 'rather than' pretty much across the board. so, if you're in doubt, you might want to go with that one.