The worst thing that happened to me was not so significant, but it still is the worst for me.
I was trying to find either pumpkins or cacti, so I left my home far away in search for at least a desert biome. I found two really neat caves, and one had a dungeon in it. I was excited... until I noticed that my inventory was full.

When I left, it started to rain, and I was struck by lightning. After that, I decided to give up on my original quest to give up on my original quest. Out of all the items that stocked my inventory, my compass was not in it.

I walk for literally an hour and a half in a huge, lifeless tundra biome and find some more caves, but no trees to create a crafting table, being out of supplies. I was about to drown myself until I saw my portal in the distance. This was the only time I was really happy to my portal for once (besides the joy of making it: "Yay, I'm gonna burn and die!").
And that's it.

EDIT: Recently while mining Obsidian, I forgot to check just one block for lava, and so when I mined some Obsidian, there was just one source block left over. I burned and tried to reach the nearby water I placed, but as you might have guessed, I screwed up, because the camera always shakes every time you take damage. The part I find the worst is that right after re-spawning, I find a pack of 5 wolves in the distance, and my 9 bones I took just in case burned with me, so I was pissed.

I even tried camping near them during the night for Skeletons to die in the morning, but they went too far and de-spawned. Sadness.