I'm wondering if one could remove their memory and store it, if it would completely remove the emotion that one felt regarding it?? With the Horcrux one removes part of their soul, usually through a violent act in order to store it within an object. With the Pensive one removes their memory and stores it within an object. That way, one can visit the memory if ever in need. If that's the case is a Pensive a dangerous tool? Could it be an explanation as to Dumbledore's general indifference?
What goes in, comes out. What you put in a Pensieve are memories and you can recover the memories from there. What you put in Horcruxes are parts of a soul and they can help rebuild the life of the person. So a Horcrux is much more advanced storage device than Pensieves (USB flash drive - as gelolopez says).
Memory: Helps you remember. Once memory is washed away by some magic (memory charm for example) the person is still alive.
Soul: Soul Keeps you alive. As we can see when dementors suck away the soul, they leave their victims in a permanent vegetative state with no life.
So if Soul is destroyed, it will have a much adverse effect.
However if memory is destroyed the person still has his life, he just won't remember things.
Which brings me to
First of all Pensieve is used to see the memories not remove them. Secondly, the memory that is put into Pensieve is not removed from the person but in fact it serves like a persistent copy of the original memory. The original memory may fade inside the person but the extracted memory retains everything from the event (of which the original memory was).
So, an extracted memory (not Pensieve) may be used to restore the original memory like horcrux can be used to restore the person but it is not as dangerous.
DISCLOSURE: No authoritative references, but carefully evaluated speculation.
Horcrux and the Pensive work different ways. In order to make a Horcrux, you have to damage your soul in order to separate the soul fragment from the rest of your soul. It is only after separating that soul fragment will you able to put the soul to an object making it a Horcrux.
Pensive on the other hand works for memories and not for soul. Dumbledore uses Pensive in order to store memory so that it won't fade. While we have the impression that Dumbledore is so intelligent it is most impossible for him to forget a memory (unless otherwise after being casted a Memory Charm), he is wise to store a piece of his memory on an outer storage device in order to keep a clear and exact idea of what the memory is. Think of Pensive as a USB flash drive.
A pensive is merely a recording device. Something that holds a memory for viewing from a third person view. A horcrux is a much more deep and sinister device. It holds a part of a person's soul. Not just memory, not just thoughts. A part of a soul would mean that person's entire persona, thoughts, memories, feelings. It is much more than just a memory. A pensive will simply display the memory that was stored. A horcrux has the ability to restore an entire persona. Also, creating a horcrux requires much more effort, the murder of a person and even possible dark magic. A pensive does not rely any of those to be recorded OR viewed.