Bit late to the party but I'll chime in. You ever see those fun visualizations of code bases as they develop? New projects coming to be, and connecting around, and being repackaged and reconnected? World-building is a lot like it, just as Adfor said. It's a great amorphous story; not the typical sort with their linear flows but more like a big web with many jumping-off points. You can choose to start wherever you want, with whatever strikes your fancy. Singular events are great if you have some in mind. I enjoy nations and geopolitics, myself, so I tend to start there. Then I'll move to conflicts, or trade, or key people within. Slowly you'll get a feel for how long you personally can spend in a given corner of your world before you need to move on, at which point you can flit among your ever-expanding web to keep it fresh to yourself. You may find that some new thing you've been building requires other parts to be redone, or better still, that the new part fits seamlessly into your web so well it's as though you'd always planned it to be there. Depending on how complex you want to make your world, and it does sound like you really want to get in there, you'll definitely want to keep notes in some way. I'm lazy, I use Notepad and Drive, but use whatever helps you plan and remember.

One last bit, and that's to say people get kind of tied up in the World vs. Story argument on this subject, right. But the fact is they go hand-in-hand, and must. A story is better for the world it plays in, and a world is at its most interesting when things are happening within it. If there's a story you want to tell, add the foundations of it to your world. If there's some aspect of your world you love and want to work on, stories will come to you in time. It's all about having fun in a sandbox only you truly know, and doing what feels right until it's time to move to a different part. Good luck!