While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

  • LuckyLu@lemmy.world
    link
    fedilink
    arrow-up
    324
    ·
    1 year ago

    Very impressed by how quickly action has been taken by this and other instances to patch the issue.

    • Darkard@lemmy.world
      link
      fedilink
      arrow-up
      20
      ·
      edit-2
      1 year ago

      Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.

      I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.

      In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

      • LuckyLu@lemmy.world
        link
        fedilink
        arrow-up
        16
        ·
        1 year ago

        I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD

        • Carnelian@lemmy.world
          link
          fedilink
          arrow-up
          12
          ·
          1 year ago

          So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear

          What I did was add my account (again)

          There was no warning or anything, and it populated the list with two of me.

          At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?

          I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.

          I logged back in, and now everything is back to normal

          • NaN@lemmy.blahaj.zone
            link
            fedilink
            arrow-up
            3
            ·
            1 year ago

            I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).

          • scarabic@lemmy.world
            link
            fedilink
            arrow-up
            2
            ·
            1 year ago

            Ah interesting. I’ve had multiple accounts from the start so it was much easier for me. Just removed my main account and added it back.

          • Pandantic@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            edit-2
            1 year ago

            I did this, but I just didn’t delete either accounts and it worked fine. Idk if it’s detrimental to have two of the same but it worked for me.

          • JJROKCZ@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Finally I found good instructions, was about to delete and reinstall until I followed this!

          • LuckyLu@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

          • LuckyLu@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

        • Infernal_pizza@lemmy.world
          link
          fedilink
          arrow-up
          7
          ·
          1 year ago

          I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it

        • calaei@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          1 year ago

          Go into account settings, clear your password, re-enter your password, save, go to feed and pull to refresh. That’s what worked for me.

        • scarabic@lemmy.world
          link
          fedilink
          arrow-up
          2
          ·
          1 year ago

          No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.

          I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”

        • No1ButtMe@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          I was I able to upvote anything or subscribe. Seems like uninstalling and reinstalling fixed my issue

        • LazaroFilm@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.

      • runswithjedi@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        Thank you so much. I was having the same issue with the website and PWA. Clearing site settings worked perfectly.

      • deweydecibel@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

        Good PSA. It took me a bit to figure it out, the app doesn’t make this obvious.

      • cookie@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        Oh, I was wondering why it was showing me as logged in but wouldn’t let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!

      • LuckyLu@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        1 year ago

        Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.

        • grissee@lemm.ee
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)

          • LuckyLu@lemmy.world
            link
            fedilink
            arrow-up
            1
            ·
            1 year ago

            Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).

            Not exactly sure I understood what you meant by “either positive or zero”.

            • grissee@lemm.ee
              link
              fedilink
              arrow-up
              2
              ·
              1 year ago

              see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)

              but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior

              for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7

              • LuckyLu@lemmy.world
                link
                fedilink
                arrow-up
                2
                ·
                1 year ago

                Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.

                That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.