I'm exactly the same way. I used to agonize over the fact that I never did the smart thing and invested or even saved on a regular basis - that my family and I would be in dire straits when I could no longer work. As it turned out, I actually was investing; it was just an investment in time, not cash. I retired toward the end of last year when I found that I could easily pay my share of the bills with money to spare - to save or spend on some small indulgence - on the income I was getting from Social Security and two pensions. (And yes, I'm aware that SSI may not be around all that much longer, but there's a very good chance it will at least outlive me.)
I did have one small bank loan that I was paying on, but when an aunt (who was a nasty, mean old woman who nobody liked) passed away and the first installment on my share of the inheritance was almost exactly enough to pay it off, I took it as a sign.
I was offered a full-time job at an attorney's office a five-minute drive from my house a couple of months ago...and I actually thought about it for a few days. But I'm learning to take it somewhat easy now, having worked nearly every day for almost fifty years, and I ended up passing on the job. My health isn't all that it could be, so I don't know how many more years I've got on this planet. It's time for me, personally, to slow it down and enjoy a quieter, more contemplative life. I have no desire to vacation in exotic places or buy hundred-thousand dollar automobiles or any of that. Sure it's fun to fantasize about something like this.
But, to coin a phrase, you can't take it with you.