I think you can blame the demise of the soap on two factors. For one, soaps are losing female viewers--their key demographic. Many women have entered the workforce and no longer have the time nor the opportunity to watch daytime television. This loss of viewers has made soaps far less palatable to advertisers.
The second factor is production costs. Of course, soaps are less costly to produce than a primetime drama or sitcom. But relative to cheapo reality fare, soaps are a pretty expensive proposition. So why spend a large sum of money on making a soap when you (the network) could spend far less to produce a talk show that would get nearly the same ratings at a fraction of the cost?