We’ve upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can’t login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn’t work.

I’ll create an issue for the devs and retry once it’s fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won’t be on 0.18.1 any time soon I’m afraid…

  • Jerti@lemmy.world
    15·
    1 year ago

    UI: 0.18.0 BE: 0.18.1-rc.1 👍

    seems to be working for me, I was already logged on

  • runswithjedi@lemmy.world
    11·
    1 year ago

    Thanks for all your work!

    Perhaps it would be beneficial to have a temporary test instance like dev.lemmy.world or something where you dedicate a limited amount of processing power to. That way you could test the update live but not affect the main instance.

    I have no idea about the specifics so that may not be doable.

    • Ruud@lemmy.worldOPM
      13·
      1 year ago

      Because of a few things, really anoying bugs and the Jerboa app not working properly with older versions

    • twistedtxb@lemmy.world
      7·
      1 year ago

      I suspect there will be a large influx of new users in two days, and that having Jerboa not working on .world might cause a few issues

    • Antik 👾@lemmy.world
      6·
      1 year ago

      For example the frontpage is no longer being constantly filled with random posts when you set the filter to “active” or “top day”.

    • dragontamer@lemmy.worldEnglish
      1·
      1 year ago

      Federation is completely borked with .18 servers. Its very difficult for us to interact with https://lemmy.ca in any way: subscribing, upvoting, comments, posting… its all bugged.

      Its maybe not that big a deal because Lemmy.world “has the most users”, so in some regards its https://lemmy.ca’s loss but… we need to restore reliable federation… especially before the July 1st rush IMO.

      The .17 to .18 upgrade is basically a soft-defederation event, because of whatever this bug is between the two versions.

    • axzxc1236@lemmy.world
      0·
      1 year ago

      Reasons I can think of:

      1. The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.

      2. Sorting is broken pre 0.18, new posts keeps flowing in.

      3. Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)

      • WhiskyTangoFoxtrot@lemmy.world
        1·
        1 year ago

        The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.

        Even that won’t work, because Jerboa initially tries to connect to lemmy.ml which is running 0.18, which older versions of Jerboa aren’t compatible with. The app just crashes instantly without giving you the opportunity to log in to your instance.

      • JdW@lemmy.world
        0·
        1 year ago

        Jerboa works fine with lemmy world, it just gives a warning and crashes on occasion. Not an issue to use it though.

        • flawedFraction@lemmy.world
          0·
          1 year ago

          For me at least, I’m unable to change the community scope or sorting method. The only thing I can use Jerboa for is “local” and sorted by “active”. It isn’t totally unusable, but I would much rather be able to switch to subscribed and hot.

        • Vanon@lemmy.world
          0·
          1 year ago

          “crashes on occasion”

          Unfortunately, the occasion seems to be at least once every five minutes for me. thisisfine.jpg This is not fine and definitely a major issue. Especially as we get closer to July 1. After this news from lemmy.world, I will have to switch, until Sync arrives. Update: Liftoff is fantastic, give it a try!

  • rrobin@lemmy.world
    6·
    1 year ago

    As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.

    We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.

    • T156@lemmy.world
      1·
      1 year ago

      Although since we seem to be rolling more than a ship in a storm, I think a proportion of lemmings would end up hospitalised for alcohol poisoning.

  • Necropola☑@lemmy.worldEnglish
    61·
    1 year ago

    Everything (UI) is extremely slow now. Not an improvement at all. Also I’m frequently getting error messages like this now:

  • snargledorf@lemmy.world
    4·
    1 year ago

    @ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

    With all the performance issues with 0.18.1, it’s highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

    • Ruud@lemmy.worldOPM
      1·
      1 year ago

      Yes, considering that. But we’ll need people to use that when we will do testing…

        • med@sh.itjust.works
          1·
          1 year ago

          One more. You find a stable way to notify anout upgrades and get a test sheet to run through and we can generate posts and activity to help test with.

          Light the beacons! lemmy.world calls for aid!

      • Finnagain@lemmy.world
        0·
        1 year ago

        I’m not familiar with what the server architecture looks like, but is there a possibility of using a load balancer in front of the instance’s server and swapping a “beta” server into the load balancer when you need to do testing? You could basically migrate your traffic with zero downtime, assuming Lemmy’s architecture allows for it.

        • Ruud@lemmy.worldOPM
          0·
          1 year ago

          Well that doesn’t differ much from what I do. I just copy the files to a second directory and test with that. Easy rollback. Downside is, that all data is lost between upgrade and rollback, which will be the same in the scenario you suggest.

          • Finnagain@lemmy.world
            1·
            1 year ago

            Ah, that’s fair. Best of luck either way. This is the tough part of admin work. FWIW, the stability we’ve had thus far has been pretty impressive!

  • Anarch157a@lemmy.world
    4·
    1 year ago

    Seems 0.18 is up… and…

    OH NO ! The interface changed ! Bring it back ! Bring it back… /s

    Just kidding… It’s looking good, so it’s time to explore a little and find all the new stuff.

    Nice work @Rudd, thanks for all the hard work.

  • OsrsNeedsF2P@lemmy.ml
    3·
    1 year ago

    Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!

    Dessalines:

    Nutomic:

    • egeres@lemmy.world
      1·
      1 year ago

      Oh god, so much coffee… it’s also thrilling to grasp a sense of what’s going on under the hood of such big social networks at a development level (not like I could understand it, but it was very interesting to see twitter’s recommendation algorithm being open-sourced)

    • imaqtpie@sh.itjust.works
      02·
      1 year ago

      Yeah they’ve been hard at work all month. But it’s also okay if things aren’t ready in time. Most of the people who matter are already here.

      Maybe we will blow up soon, maybe later, but the quality of content here is sufficient to drive growth regardless of whether or not we get the prophesized huge migrations from reddit

      • thegreatgarbo@lemmy.world
        1·
        1 year ago

        Agreed. I’m here to stay with the content from the last couple weeks. 3 day poop embargoes alone is worth it. And recognizing users because the community is still small enough is SO different from they who shall not be named.

  • necrxfagivs@lemmy.world
    3·
    1 year ago

    Great! I’m getting a bug, when I upvote a comment or post, the “loading animation” takes forever to complete. If I refresh the upvote is there. It reminds me to the bug we had with comments and new posts.

    Edit: Is the same for comments.