Bump version

* generate fallback web archive URL if none exists

* remove fallback web archive snapshot creation

* fix test

.gitignore

@@ -191,3 +191,8 @@ typings/
 /tmp
 # Database file
 /data
+# ublock + chromium
+/uBlock0.chromium
+/chromium-profile
+# direnv
+/.direnv

CHANGELOG.md

@@ -1,5 +1,74 @@
 # Changelog
 
+## v1.31.0 (16/06/2024)
+
+### What's Changed
+* Add support for bookmark thumbnails by @vslinko in https://github.com/sissbruecker/linkding/pull/721
+* Automatically add tags to bookmarks based on URL pattern by @vslinko in https://github.com/sissbruecker/linkding/pull/736
+* Load bookmark thumbnails after import by @vslinko in https://github.com/sissbruecker/linkding/pull/724
+* Load missing thumbnails after enabling the feature by @sissbruecker in https://github.com/sissbruecker/linkding/pull/725
+* Thumbnails lazy loading by @vslinko in https://github.com/sissbruecker/linkding/pull/734
+* Add option for disabling tag grouping by @vslinko in https://github.com/sissbruecker/linkding/pull/735
+* Preview auto tags in bookmark form by @sissbruecker in https://github.com/sissbruecker/linkding/pull/737
+* Hide tooltip on mobile by @vslinko in https://github.com/sissbruecker/linkding/pull/733
+* Bump requests from 2.31.0 to 2.32.0 by @dependabot in https://github.com/sissbruecker/linkding/pull/740
+
+### New Contributors
+* @vslinko made their first contribution in https://github.com/sissbruecker/linkding/pull/721
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.30.0...v1.31.0
+
+---
+
+## v1.30.0 (20/04/2024)
+
+### What's Changed
+* Add reader mode by @sissbruecker in https://github.com/sissbruecker/linkding/pull/703
+* Allow uploading custom files for bookmarks by @sissbruecker in https://github.com/sissbruecker/linkding/pull/713
+* Add option for marking bookmarks as unread by default by @ab623 in https://github.com/sissbruecker/linkding/pull/706
+* Make blocking cookie banners more reliable by @sissbruecker in https://github.com/sissbruecker/linkding/pull/699
+* Close bookmark details with escape by @sissbruecker in https://github.com/sissbruecker/linkding/pull/702
+* Show proper name for bookmark assets in admin by @ab623 in https://github.com/sissbruecker/linkding/pull/708
+* Bump sqlparse from 0.4.4 to 0.5.0 by @dependabot in https://github.com/sissbruecker/linkding/pull/704
+
+### New Contributors
+* @ab623 made their first contribution in https://github.com/sissbruecker/linkding/pull/706
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.29.0...v1.30.0
+
+---
+
+## v1.29.0 (14/04/2024)
+
+### What's Changed
+* Remove ads and cookie banners from HTML snapshots by @sissbruecker in https://github.com/sissbruecker/linkding/pull/695
+* Add button for creating missing HTML snapshots by @sissbruecker in https://github.com/sissbruecker/linkding/pull/696
+* Refresh file list when there are queued snapshots by @sissbruecker in https://github.com/sissbruecker/linkding/pull/697
+* Bump idna from 3.6 to 3.7 by @dependabot in https://github.com/sissbruecker/linkding/pull/694
+
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.28.0...v1.29.0
+
+---
+
+## v1.28.0 (09/04/2024)
+
+### What's Changed
+* Add option to disable SSL verification for OIDC by @akaSyntaax in https://github.com/sissbruecker/linkding/pull/684
+* Add full backup method by @sissbruecker in https://github.com/sissbruecker/linkding/pull/686
+* Truncate snapshot filename for long URLs by @sissbruecker in https://github.com/sissbruecker/linkding/pull/687
+* Add option for customizing single-file timeout by @pettijohn in https://github.com/sissbruecker/linkding/pull/688
+* Add option for passing arguments to single-file command by @pettijohn in https://github.com/sissbruecker/linkding/pull/691
+* Fix typo by @tianheg in https://github.com/sissbruecker/linkding/pull/689
+
+### New Contributors
+* @akaSyntaax made their first contribution in https://github.com/sissbruecker/linkding/pull/684
+* @pettijohn made their first contribution in https://github.com/sissbruecker/linkding/pull/688
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.27.1...v1.28.0
+
+---
+
 ## v1.27.1 (07/04/2024)
 
 ### What's Changed

Makefile

@@ -7,7 +7,7 @@ tasks:
 	python manage.py process_tasks
 
 test:
-	pytest
+	pytest -n auto
 
 format:
 	black bookmarks

README.md

@@ -45,7 +45,7 @@ The name comes from:
 - Admin panel for user self-service and raw data access
 
 
-**Demo:** https://demo.linkding.link/ ([see here](https://github.com/sissbruecker/linkding/issues/408) if you have trouble accessing it)
+**Demo:** https://demo.linkding.link/
 
 **Screenshot:**
 
@@ -237,6 +237,7 @@ This section lists community projects around using linkding, in alphabetical ord
 - [Linkdy](https://github.com/JGeek00/linkdy): An open source mobile and desktop (not yet) client created with Flutter. Available at the [Google Play Store](https://play.google.com/store/apps/details?id=com.jgeek00.linkdy). By [JGeek00](https://github.com/JGeek00).
 - [LinkThing](https://apps.apple.com/us/app/linkthing/id1666031776) An iOS client for linkding. By [amoscardino](https://github.com/amoscardino)
 - [Open all links bookmarklet](https://gist.github.com/ukcuddlyguy/336dd7339e6d35fc64a75ccfc9323c66) A browser bookmarklet to open all links on the current Linkding page in new tabs. By [ukcuddlyguy](https://github.com/ukcuddlyguy)
+- [Pinkt](https://github.com/fibelatti/pinboard-kotlin) An Android client for linkding. By [fibelatti](https://github.com/fibelatti)
 - [Postman collection](https://gist.github.com/gingerbeardman/f0b42502f3bc9344e92ce63afd4360d3) a group of saved request templates for API testing. By [gingerbeardman](https://github.com/gingerbeardman)
 
 ## Acknowledgements + Donations

assets/logo.svg

@@ -0,0 +1 @@
+<svg clip-rule="evenodd" fill-rule="evenodd" height="512" stroke-linejoin="round" stroke-miterlimit="1.5" viewBox="0 0 512 512" width="512" xmlns="http://www.w3.org/2000/svg"><circle cx="255.0164" cy="254.9236" fill="#5856e0" r="224.78528" stroke-width="1.18"/><g fill="none" stroke="#fff" stroke-width="31.25"><path d="m1244.39 1293.95v199.64s-.81 67.89 74.9 68.88c75.98.99 74.88-68.88 74.88-68.88v-199.64" transform="matrix(.70710678 .70710678 -.70710678 .70710678 284.139117 -1684.198509)"/><path d="m1244.39 1293.95v199.64s-.81 67.89 74.9 68.88c75.98.99 74.88-68.88 74.88-68.88v-199.64" transform="matrix(-.70957074 -.70463421 .70463421 -.70957074 235.113139 2195.434643)"/></g></svg>

bookmarks/admin.py

@@ -200,9 +200,13 @@ class AdminBookmark(admin.ModelAdmin):
 
 
 class AdminBookmarkAsset(admin.ModelAdmin):
-    list_display = ("display_name", "date_created", "status")
+    @admin.display(description="Display Name")
+    def custom_display_name(self, obj):
+        return str(obj)
+
+    list_display = ("custom_display_name", "date_created", "status")
     search_fields = (
-        "display_name",
+        "custom_display_name",
         "file",
     )
     list_filter = ("status",)

bookmarks/api/routes.py

@@ -11,6 +11,7 @@ from bookmarks.api.serializers import (
     UserProfileSerializer,
 )
 from bookmarks.models import Bookmark, BookmarkSearch, Tag, User
+from bookmarks.services import auto_tagging
 from bookmarks.services.bookmarks import (
     archive_bookmark,
     unarchive_bookmark,
@@ -51,7 +52,7 @@ class BookmarkViewSet(
         return Bookmark.objects.all().filter(owner=user)
 
     def get_serializer_context(self):
-        return {"user": self.request.user}
+        return {"request": self.request, "user": self.request.user}
 
     @action(methods=["get"], detail=False)
     def archived(self, request):
@@ -59,8 +60,8 @@ class BookmarkViewSet(
         search = BookmarkSearch.from_request(request.GET)
         query_set = queries.query_archived_bookmarks(user, user.profile, search)
         page = self.paginate_queryset(query_set)
-        serializer = self.get_serializer_class()
-        data = serializer(page, many=True).data
+        serializer = self.get_serializer(page, many=True)
+        data = serializer.data
         return self.get_paginated_response(data)
 
     @action(methods=["get"], detail=False)
@@ -72,8 +73,8 @@ class BookmarkViewSet(
             user, request.user_profile, search, public_only
         )
         page = self.paginate_queryset(query_set)
-        serializer = self.get_serializer_class()
-        data = serializer(page, many=True).data
+        serializer = self.get_serializer(page, many=True)
+        data = serializer.data
         return self.get_paginated_response(data)
 
     @action(methods=["post"], detail=True)
@@ -99,13 +100,26 @@ class BookmarkViewSet(
         # Either return metadata from existing bookmark, or scrape from URL
         if bookmark:
             metadata = WebsiteMetadata(
-                url, bookmark.website_title, bookmark.website_description
+                url,
+                bookmark.website_title,
+                bookmark.website_description,
+                None,
             )
         else:
             metadata = website_loader.load_website_metadata(url)
 
+        # Return tags that would be automatically applied to the bookmark
+        profile = request.user.profile
+        auto_tags = []
+        if profile.auto_tagging_rules:
+            auto_tags = auto_tagging.get_tags(profile.auto_tagging_rules, url)
+
         return Response(
-            {"bookmark": existing_bookmark_data, "metadata": metadata.to_dict()},
+            {
+                "bookmark": existing_bookmark_data,
+                "metadata": metadata.to_dict(),
+                "auto_tags": auto_tags,
+            },
             status=status.HTTP_200_OK,
         )
 

bookmarks/api/serializers.py

@@ -1,4 +1,5 @@
 from django.db.models import prefetch_related_objects
+from django.templatetags.static import static
 from rest_framework import serializers
 from rest_framework.serializers import ListSerializer
 
@@ -31,6 +32,8 @@ class BookmarkSerializer(serializers.ModelSerializer):
             "website_title",
             "website_description",
             "web_archive_snapshot_url",
+            "favicon_url",
+            "preview_image_url",
             "is_archived",
             "unread",
             "shared",
@@ -42,6 +45,8 @@ class BookmarkSerializer(serializers.ModelSerializer):
             "website_title",
             "website_description",
             "web_archive_snapshot_url",
+            "favicon_url",
+            "preview_image_url",
             "date_added",
             "date_modified",
         ]
@@ -56,6 +61,24 @@ class BookmarkSerializer(serializers.ModelSerializer):
     shared = serializers.BooleanField(required=False, default=False)
     # Override readonly tag_names property to allow passing a list of tag names to create/update
     tag_names = TagListField(required=False, default=[])
+    favicon_url = serializers.SerializerMethodField()
+    preview_image_url = serializers.SerializerMethodField()
+
+    def get_favicon_url(self, obj: Bookmark):
+        if not obj.favicon_file:
+            return None
+        request = self.context.get("request")
+        favicon_file_path = static(obj.favicon_file)
+        favicon_url = request.build_absolute_uri(favicon_file_path)
+        return favicon_url
+
+    def get_preview_image_url(self, obj: Bookmark):
+        if not obj.preview_image_file:
+            return None
+        request = self.context.get("request")
+        preview_image_file_path = static(obj.preview_image_file)
+        preview_image_url = request.build_absolute_uri(preview_image_file_path)
+        return preview_image_url
 
     def create(self, validated_data):
         bookmark = Bookmark()

bookmarks/e2e/e2e_test_bookmark_details_modal.py

@@ -1,3 +1,4 @@
+from django.test import override_settings
 from django.urls import reverse
 from playwright.sync_api import sync_playwright, expect
 
@@ -33,6 +34,11 @@ class BookmarkDetailsModalE2ETestCase(LinkdingE2ETestCase):
             overlay.click(position={"x": 0, "y": 0})
             expect(details_modal).to_be_hidden()
 
+            # close with escape
+            details_modal = self.open_details_modal(bookmark)
+            self.page.keyboard.press("Escape")
+            expect(details_modal).to_be_hidden()
+
     def test_toggle_archived(self):
         bookmark = self.setup_bookmark()
 
@@ -44,14 +50,17 @@ class BookmarkDetailsModalE2ETestCase(LinkdingE2ETestCase):
             details_modal = self.open_details_modal(bookmark)
             details_modal.get_by_text("Archived", exact=False).click()
             expect(self.locate_bookmark(bookmark.title)).not_to_be_visible()
+            self.assertReloads(0)
 
             # unarchive
             url = reverse("bookmarks:archived")
             self.page.goto(self.live_server_url + url)
+            self.resetReloads()
 
             details_modal = self.open_details_modal(bookmark)
             details_modal.get_by_text("Archived", exact=False).click()
             expect(self.locate_bookmark(bookmark.title)).not_to_be_visible()
+            self.assertReloads(0)
 
     def test_toggle_unread(self):
         bookmark = self.setup_bookmark()
@@ -66,11 +75,13 @@ class BookmarkDetailsModalE2ETestCase(LinkdingE2ETestCase):
             details_modal.get_by_text("Unread").click()
             bookmark_item = self.locate_bookmark(bookmark.title)
             expect(bookmark_item.get_by_text("Unread")).to_be_visible()
+            self.assertReloads(0)
 
             # mark as read
             details_modal.get_by_text("Unread").click()
             bookmark_item = self.locate_bookmark(bookmark.title)
             expect(bookmark_item.get_by_text("Unread")).not_to_be_visible()
+            self.assertReloads(0)
 
     def test_toggle_shared(self):
         profile = self.get_or_create_test_user().profile
@@ -89,11 +100,13 @@ class BookmarkDetailsModalE2ETestCase(LinkdingE2ETestCase):
             details_modal.get_by_text("Shared").click()
             bookmark_item = self.locate_bookmark(bookmark.title)
             expect(bookmark_item.get_by_text("Shared")).to_be_visible()
+            self.assertReloads(0)
 
             # unshare bookmark
             details_modal.get_by_text("Shared").click()
             bookmark_item = self.locate_bookmark(bookmark.title)
             expect(bookmark_item.get_by_text("Shared")).not_to_be_visible()
+            self.assertReloads(0)
 
     def test_edit_return_url(self):
         bookmark = self.setup_bookmark()
@@ -131,3 +144,33 @@ class BookmarkDetailsModalE2ETestCase(LinkdingE2ETestCase):
             expect(self.locate_bookmark(bookmark.title)).not_to_be_visible()
 
         self.assertEqual(Bookmark.objects.count(), 0)
+
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_create_snapshot_remove_snapshot(self):
+        bookmark = self.setup_bookmark()
+
+        with sync_playwright() as p:
+            url = reverse("bookmarks:index") + f"?q={bookmark.title}"
+            self.open(url, p)
+
+            details_modal = self.open_details_modal(bookmark)
+            asset_list = details_modal.locator(".assets")
+
+            # No snapshots initially
+            snapshot = asset_list.get_by_text("HTML snapshot from", exact=False)
+            expect(snapshot).not_to_be_visible()
+
+            # Create snapshot
+            details_modal.get_by_text("Create HTML snapshot", exact=False).click()
+            self.assertReloads(0)
+
+            # Has new snapshots
+            expect(snapshot).to_be_visible()
+
+            # Create snapshot
+            asset_list.get_by_text("Remove", exact=False).click()
+            asset_list.get_by_text("Confirm", exact=False).click()
+
+            # Snapshot is removed
+            expect(snapshot).not_to_be_visible()
+            self.assertReloads(0)

bookmarks/e2e/e2e_test_bookmark_form.py

@@ -85,3 +85,25 @@ class BookmarkFormE2ETestCase(LinkdingE2ETestCase):
 
             page.get_by_label("URL").fill(bookmark.url)
             expect(details).to_have_attribute("open", value="")
+
+    def test_create_should_preview_auto_tags(self):
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = "github.com dev github"
+        profile.save()
+
+        with sync_playwright() as p:
+            # Open page with URL that should have auto tags
+            browser = self.setup_browser(p)
+            page = browser.new_page()
+            url = self.live_server_url + reverse("bookmarks:new")
+            url += f"?url=https%3A%2F%2Fgithub.com%2Fsissbruecker%2Flinkding"
+            page.goto(url)
+
+            auto_tags_hint = page.locator(".form-input-hint.auto-tags")
+            expect(auto_tags_hint).to_be_visible()
+            expect(auto_tags_hint).to_have_text("Auto tags: dev github")
+
+            # Change to URL without auto tags
+            page.get_by_label("URL").fill("https://example.com")
+
+            expect(auto_tags_hint).to_be_hidden()

bookmarks/e2e/e2e_test_bookmark_page_bulk_edit.py

@@ -194,7 +194,7 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
 
             self.locate_bulk_edit_toggle().click()
 
-            checkboxes = page.locator("label[ld-bulk-edit-checkbox] input")
+            checkboxes = page.locator("label.bulk-edit-checkbox input")
             self.assertEqual(6, checkboxes.count())
             for i in range(checkboxes.count()):
                 expect(checkboxes.nth(i)).not_to_be_checked()
@@ -264,13 +264,13 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
 
             # Hide select across by toggling a single bookmark
             self.locate_bookmark("Bookmark 1").locator(
-                "label[ld-bulk-edit-checkbox]"
+                "label.bulk-edit-checkbox"
             ).click()
             expect(self.locate_bulk_edit_select_across()).not_to_be_visible()
 
             # Show select across again, verify it is unchecked
             self.locate_bookmark("Bookmark 1").locator(
-                "label[ld-bulk-edit-checkbox]"
+                "label.bulk-edit-checkbox"
             ).click()
             expect(self.locate_bulk_edit_select_across()).not_to_be_checked()
 
@@ -297,7 +297,7 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
             expect(bookmark_list).not_to_be_visible()
 
             # Verify bulk edit checkboxes are reset
-            checkboxes = page.locator("label[ld-bulk-edit-checkbox] input")
+            checkboxes = page.locator("label.bulk-edit-checkbox input")
             self.assertEqual(31, checkboxes.count())
             for i in range(checkboxes.count()):
                 expect(checkboxes.nth(i)).not_to_be_checked()

bookmarks/e2e/e2e_test_bookmark_page_partial_updates.py

@@ -169,7 +169,7 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
 
             self.locate_bulk_edit_toggle().click()
             self.locate_bookmark("Bookmark 2").locator(
-                "label[ld-bulk-edit-checkbox]"
+                "label.bulk-edit-checkbox"
             ).click()
             self.select_bulk_action("Archive")
             self.locate_bulk_edit_bar().get_by_text("Execute").click()
@@ -187,7 +187,7 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
 
             self.locate_bulk_edit_toggle().click()
             self.locate_bookmark("Bookmark 2").locator(
-                "label[ld-bulk-edit-checkbox]"
+                "label.bulk-edit-checkbox"
             ).click()
             self.select_bulk_action("Delete")
             self.locate_bulk_edit_bar().get_by_text("Execute").click()
@@ -230,7 +230,7 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
 
             self.locate_bulk_edit_toggle().click()
             self.locate_bookmark("Archived Bookmark 2").locator(
-                "label[ld-bulk-edit-checkbox]"
+                "label.bulk-edit-checkbox"
             ).click()
             self.select_bulk_action("Unarchive")
             self.locate_bulk_edit_bar().get_by_text("Execute").click()
@@ -248,7 +248,7 @@ class BookmarkPagePartialUpdatesE2ETestCase(LinkdingE2ETestCase):
 
             self.locate_bulk_edit_toggle().click()
             self.locate_bookmark("Archived Bookmark 2").locator(
-                "label[ld-bulk-edit-checkbox]"
+                "label.bulk-edit-checkbox"
             ).click()
             self.select_bulk_action("Delete")
             self.locate_bulk_edit_bar().get_by_text("Execute").click()

bookmarks/e2e/e2e_test_tag_cloud_modal.py

@@ -0,0 +1,76 @@
+from django.test import override_settings
+from django.urls import reverse
+from playwright.sync_api import sync_playwright, expect, Locator
+
+from bookmarks.e2e.helpers import LinkdingE2ETestCase
+from bookmarks.models import Bookmark
+
+
+class TagCloudModalE2ETestCase(LinkdingE2ETestCase):
+    def test_show_modal_close_modal(self):
+        self.setup_bookmark(tags=[self.setup_tag(name="cooking")])
+        self.setup_bookmark(tags=[self.setup_tag(name="hiking")])
+
+        with sync_playwright() as p:
+            page = self.open(reverse("bookmarks:index"), p)
+
+            # use smaller viewport to make tags button visible
+            page.set_viewport_size({"width": 375, "height": 812})
+
+            # open tag cloud modal
+            modal_trigger = page.locator(".content-area-header").get_by_role(
+                "button", name="Tags"
+            )
+            modal_trigger.click()
+
+            # verify modal is visible
+            modal = page.locator(".modal")
+            expect(modal).to_be_visible()
+            expect(modal.locator(".modal-title")).to_have_text("Tags")
+
+            # close with close button
+            modal.locator("button.close").click()
+            expect(modal).to_be_hidden()
+
+            # open modal again
+            modal_trigger.click()
+
+            # close with backdrop
+            backdrop = modal.locator(".modal-overlay")
+            backdrop.click(position={"x": 0, "y": 0})
+            expect(modal).to_be_hidden()
+
+    def test_select_tag(self):
+        self.setup_bookmark(tags=[self.setup_tag(name="cooking")])
+        self.setup_bookmark(tags=[self.setup_tag(name="hiking")])
+
+        with sync_playwright() as p:
+            page = self.open(reverse("bookmarks:index"), p)
+
+            # use smaller viewport to make tags button visible
+            page.set_viewport_size({"width": 375, "height": 812})
+
+            # open tag cloud modal
+            modal_trigger = page.locator(".content-area-header").get_by_role(
+                "button", name="Tags"
+            )
+            modal_trigger.click()
+
+            # verify tags are displayed
+            modal = page.locator(".modal")
+            unselected_tags = modal.locator(".unselected-tags")
+            expect(unselected_tags.get_by_text("cooking")).to_be_visible()
+            expect(unselected_tags.get_by_text("hiking")).to_be_visible()
+
+            # select tag
+            unselected_tags.get_by_text("cooking").click()
+
+            # open modal again
+            modal_trigger.click()
+
+            # verify tag is selected, other tag is not visible anymore
+            selected_tags = modal.locator(".selected-tags")
+            expect(selected_tags.get_by_text("cooking")).to_be_visible()
+
+            expect(unselected_tags.get_by_text("cooking")).not_to_be_visible()
+            expect(unselected_tags.get_by_text("hiking")).not_to_be_visible()

bookmarks/e2e/helpers.py

@@ -39,6 +39,9 @@ class LinkdingE2ETestCase(LiveServerTestCase, BookmarkFactoryMixin):
     def assertReloads(self, count: int):
         self.assertEqual(self.num_loads, count)
 
+    def resetReloads(self):
+        self.num_loads = 0
+
     def locate_bookmark_list(self):
         return self.page.locator("ul[ld-bookmark-list]")
 
@@ -62,7 +65,7 @@ class LinkdingE2ETestCase(LiveServerTestCase, BookmarkFactoryMixin):
         return self.page.locator(".bulk-edit-bar")
 
     def locate_bulk_edit_select_all(self):
-        return self.locate_bulk_edit_bar().locator("label[ld-bulk-edit-checkbox][all]")
+        return self.locate_bulk_edit_bar().locator("label.bulk-edit-checkbox.all")
 
     def locate_bulk_edit_select_across(self):
         return self.locate_bulk_edit_bar().locator("label.select-across")

bookmarks/frontend/behaviors/bookmark-page.js

@@ -1,67 +1,8 @@
-import { registerBehavior, swapContent } from "./index";
+import { Behavior, registerBehavior } from "./index";
 
-class BookmarkPage {
+class BookmarkItem extends Behavior {
   constructor(element) {
-    this.element = element;
-    this.form = element.querySelector("form.bookmark-actions");
-    this.form.addEventListener("submit", this.onFormSubmit.bind(this));
-
-    this.bookmarkList = element.querySelector(".bookmark-list-container");
-    this.tagCloud = element.querySelector(".tag-cloud-container");
-
-    document.addEventListener("bookmark-page-refresh", () => {
-      this.refresh();
-    });
-  }
-
-  async onFormSubmit(event) {
-    event.preventDefault();
-
-    const url = this.form.action;
-    const formData = new FormData(this.form);
-    formData.append(event.submitter.name, event.submitter.value);
-
-    await fetch(url, {
-      method: "POST",
-      body: formData,
-      redirect: "manual", // ignore redirect
-    });
-    await this.refresh();
-  }
-
-  async refresh() {
-    const query = window.location.search;
-    const bookmarksUrl = this.element.getAttribute("bookmarks-url");
-    const tagsUrl = this.element.getAttribute("tags-url");
-    Promise.all([
-      fetch(`${bookmarksUrl}${query}`).then((response) => response.text()),
-      fetch(`${tagsUrl}${query}`).then((response) => response.text()),
-    ]).then(([bookmarkListHtml, tagCloudHtml]) => {
-      swapContent(this.bookmarkList, bookmarkListHtml);
-      swapContent(this.tagCloud, tagCloudHtml);
-
-      // Dispatch list updated event
-      const listElement = this.bookmarkList.querySelector(
-        "ul[data-bookmarks-total]",
-      );
-      const bookmarksTotal =
-        (listElement && listElement.dataset.bookmarksTotal) || 0;
-
-      this.bookmarkList.dispatchEvent(
-        new CustomEvent("bookmark-list-updated", {
-          bubbles: true,
-          detail: { bookmarksTotal },
-        }),
-      );
-    });
-  }
-}
-
-registerBehavior("ld-bookmark-page", BookmarkPage);
-
-class BookmarkItem {
-  constructor(element) {
-    this.element = element;
+    super(element);
 
     // Toggle notes
     const notesToggle = element.querySelector(".toggle-notes");
@@ -72,9 +13,11 @@ class BookmarkItem {
     // Add tooltip to title if it is truncated
     const titleAnchor = element.querySelector(".title > a");
     const titleSpan = titleAnchor.querySelector("span");
-    if (titleSpan.offsetWidth > titleAnchor.offsetWidth) {
-      titleAnchor.dataset.tooltip = titleSpan.textContent;
-    }
+    requestAnimationFrame(() => {
+      if (titleSpan.offsetWidth > titleAnchor.offsetWidth) {
+        titleAnchor.dataset.tooltip = titleSpan.textContent;
+      }
+    });
   }
 
   onToggleNotes(event) {

bookmarks/frontend/behaviors/bulk-edit.js

@@ -1,46 +1,60 @@
-import { registerBehavior } from "./index";
+import { Behavior, registerBehavior } from "./index";
 
-class BulkEdit {
+class BulkEdit extends Behavior {
   constructor(element) {
-    this.element = element;
+    super(element);
+
     this.active = false;
-    this.actionSelect = element.querySelector("select[name='bulk_action']");
-    this.tagAutoComplete = element.querySelector(".tag-autocomplete");
-    this.selectAcross = element.querySelector("label.select-across");
 
-    element.addEventListener(
-      "bulk-edit-toggle-active",
-      this.onToggleActive.bind(this),
-    );
-    element.addEventListener(
-      "bulk-edit-toggle-all",
-      this.onToggleAll.bind(this),
-    );
-    element.addEventListener(
-      "bulk-edit-toggle-bookmark",
-      this.onToggleBookmark.bind(this),
-    );
-    element.addEventListener(
-      "bookmark-list-updated",
-      this.onListUpdated.bind(this),
-    );
+    this.onToggleActive = this.onToggleActive.bind(this);
+    this.onToggleAll = this.onToggleAll.bind(this);
+    this.onToggleBookmark = this.onToggleBookmark.bind(this);
+    this.onActionSelected = this.onActionSelected.bind(this);
 
-    this.actionSelect.addEventListener(
-      "change",
-      this.onActionSelected.bind(this),
-    );
+    this.init();
+    // Reset when bookmarks are refreshed
+    document.addEventListener("refresh-bookmark-list-done", () => this.init());
   }
 
-  get allCheckbox() {
-    return this.element.querySelector("[ld-bulk-edit-checkbox][all] input");
-  }
+  init() {
+    // Update elements
+    this.activeToggle = this.element.querySelector(".bulk-edit-active-toggle");
+    this.actionSelect = this.element.querySelector(
+      "select[name='bulk_action']",
+    );
+    this.tagAutoComplete = this.element.querySelector(".tag-autocomplete");
+    this.selectAcross = this.element.querySelector("label.select-across");
+    this.allCheckbox = this.element.querySelector(
+      ".bulk-edit-checkbox.all input",
+    );
+    this.bookmarkCheckboxes = Array.from(
+      this.element.querySelectorAll(".bulk-edit-checkbox:not(.all) input"),
+    );
 
-  get bookmarkCheckboxes() {
-    return [
-      ...this.element.querySelectorAll(
-        "[ld-bulk-edit-checkbox]:not([all]) input",
-      ),
-    ];
+    // Remove previous listeners if elements are the same
+    this.activeToggle.removeEventListener("click", this.onToggleActive);
+    this.actionSelect.removeEventListener("change", this.onActionSelected);
+    this.allCheckbox.removeEventListener("change", this.onToggleAll);
+    this.bookmarkCheckboxes.forEach((checkbox) => {
+      checkbox.removeEventListener("change", this.onToggleBookmark);
+    });
+
+    // Reset checkbox states
+    this.reset();
+
+    // Update total number of bookmarks
+    const totalHolder = this.element.querySelector("[data-bookmarks-total]");
+    const total = totalHolder?.dataset.bookmarksTotal || 0;
+    const totalSpan = this.selectAcross.querySelector("span.total");
+    totalSpan.textContent = total;
+
+    // Add new listeners
+    this.activeToggle.addEventListener("click", this.onToggleActive);
+    this.actionSelect.addEventListener("change", this.onActionSelected);
+    this.allCheckbox.addEventListener("change", this.onToggleAll);
+    this.bookmarkCheckboxes.forEach((checkbox) => {
+      checkbox.addEventListener("change", this.onToggleBookmark);
+    });
   }
 
   onToggleActive() {
@@ -81,16 +95,6 @@ class BulkEdit {
     }
   }
 
-  onListUpdated(event) {
-    // Reset checkbox states
-    this.reset();
-
-    // Update total number of bookmarks
-    const total = event.detail.bookmarksTotal;
-    const totalSpan = this.selectAcross.querySelector("span.total");
-    totalSpan.textContent = total;
-  }
-
   updateSelectAcross(allChecked) {
     if (allChecked) {
       this.selectAcross.classList.remove("d-none");
@@ -109,33 +113,4 @@ class BulkEdit {
   }
 }
 
-class BulkEditActiveToggle {
-  constructor(element) {
-    this.element = element;
-    element.addEventListener("click", this.onClick.bind(this));
-  }
-
-  onClick() {
-    this.element.dispatchEvent(
-      new CustomEvent("bulk-edit-toggle-active", { bubbles: true }),
-    );
-  }
-}
-
-class BulkEditCheckbox {
-  constructor(element) {
-    this.element = element;
-    element.addEventListener("change", this.onChange.bind(this));
-  }
-
-  onChange() {
-    const type = this.element.hasAttribute("all") ? "all" : "bookmark";
-    this.element.dispatchEvent(
-      new CustomEvent(`bulk-edit-toggle-${type}`, { bubbles: true }),
-    );
-  }
-}
-
 registerBehavior("ld-bulk-edit", BulkEdit);
-registerBehavior("ld-bulk-edit-active-toggle", BulkEditActiveToggle);
-registerBehavior("ld-bulk-edit-checkbox", BulkEditCheckbox);

bookmarks/frontend/behaviors/confirm-button.js

@@ -1,25 +1,29 @@
-import { registerBehavior } from "./index";
+import { Behavior, registerBehavior } from "./index";
 
-class ConfirmButtonBehavior {
+class ConfirmButtonBehavior extends Behavior {
   constructor(element) {
-    const button = element;
-    button.dataset.type = button.type;
-    button.dataset.name = button.name;
-    button.dataset.value = button.value;
-    button.removeAttribute("type");
-    button.removeAttribute("name");
-    button.removeAttribute("value");
-    button.addEventListener("click", this.onClick.bind(this));
-    this.button = button;
+    super(element);
+    element.dataset.type = element.type;
+    element.dataset.name = element.name;
+    element.dataset.value = element.value;
+    element.removeAttribute("type");
+    element.removeAttribute("name");
+    element.removeAttribute("value");
+    element.addEventListener("click", this.onClick.bind(this));
+  }
+
+  destroy() {
+    Behavior.interacting = false;
   }
 
   onClick(event) {
     event.preventDefault();
+    Behavior.interacting = true;
 
     const container = document.createElement("span");
     container.className = "confirmation";
 
-    const icon = this.button.getAttribute("confirm-icon");
+    const icon = this.element.getAttribute("ld-confirm-icon");
     if (icon) {
       const iconElement = document.createElementNS(
         "http://www.w3.org/2000/svg",
@@ -31,27 +35,27 @@ class ConfirmButtonBehavior {
       container.append(iconElement);
     }
 
-    const question = this.button.getAttribute("confirm-question");
+    const question = this.element.getAttribute("ld-confirm-question");
     if (question) {
       const questionElement = document.createElement("span");
       questionElement.innerText = question;
       container.append(question);
     }
 
-    const buttonClasses = Array.from(this.button.classList.values())
+    const buttonClasses = Array.from(this.element.classList.values())
       .filter((cls) => cls.startsWith("btn"))
       .join(" ");
 
-    const cancelButton = document.createElement(this.button.nodeName);
+    const cancelButton = document.createElement(this.element.nodeName);
     cancelButton.type = "button";
     cancelButton.innerText = question ? "No" : "Cancel";
     cancelButton.className = `${buttonClasses} mr-1`;
     cancelButton.addEventListener("click", this.reset.bind(this));
 
-    const confirmButton = document.createElement(this.button.nodeName);
-    confirmButton.type = this.button.dataset.type;
-    confirmButton.name = this.button.dataset.name;
-    confirmButton.value = this.button.dataset.value;
+    const confirmButton = document.createElement(this.element.nodeName);
+    confirmButton.type = this.element.dataset.type;
+    confirmButton.name = this.element.dataset.name;
+    confirmButton.value = this.element.dataset.value;
     confirmButton.innerText = question ? "Yes" : "Confirm";
     confirmButton.className = buttonClasses;
     confirmButton.addEventListener("click", this.reset.bind(this));
@@ -59,14 +63,15 @@ class ConfirmButtonBehavior {
     container.append(cancelButton, confirmButton);
     this.container = container;
 
-    this.button.before(container);
-    this.button.classList.add("d-none");
+    this.element.before(container);
+    this.element.classList.add("d-none");
   }
 
   reset() {
     setTimeout(() => {
+      Behavior.interacting = false;
       this.container.remove();
-      this.button.classList.remove("d-none");
+      this.element.classList.remove("d-none");
     });
   }
 }

bookmarks/frontend/behaviors/dropdown.js

@@ -1,8 +1,8 @@
-import { registerBehavior } from "./index";
+import { Behavior, registerBehavior } from "./index";
 
-class DropdownBehavior {
+class DropdownBehavior extends Behavior {
   constructor(element) {
-    this.element = element;
+    super(element);
     this.opened = false;
     this.onOutsideClick = this.onOutsideClick.bind(this);
 

bookmarks/frontend/behaviors/fetch.js

@@ -0,0 +1,48 @@
+import { Behavior, fireEvents, registerBehavior, swap } from "./index";
+
+class FetchBehavior extends Behavior {
+  constructor(element) {
+    super(element);
+
+    const eventName = element.getAttribute("ld-on");
+    const interval = parseInt(element.getAttribute("ld-interval")) * 1000;
+
+    this.onFetch = this.onFetch.bind(this);
+    this.onInterval = this.onInterval.bind(this);
+
+    element.addEventListener(eventName, this.onFetch);
+    if (interval) {
+      this.intervalId = setInterval(this.onInterval, interval);
+    }
+  }
+
+  destroy() {
+    if (this.intervalId) {
+      clearInterval(this.intervalId);
+    }
+  }
+
+  async onFetch(maybeEvent) {
+    if (maybeEvent) {
+      maybeEvent.preventDefault();
+    }
+    const url = this.element.getAttribute("ld-fetch");
+    const html = await fetch(url).then((response) => response.text());
+
+    const target = this.element.getAttribute("ld-target");
+    const select = this.element.getAttribute("ld-select");
+    swap(this.element, html, { target, select });
+
+    const events = this.element.getAttribute("ld-fire");
+    fireEvents(events);
+  }
+
+  onInterval() {
+    if (Behavior.interacting) {
+      return;
+    }
+    this.onFetch();
+  }
+}
+
+registerBehavior("ld-fetch", FetchBehavior);

bookmarks/frontend/behaviors/form.js

@@ -1,12 +1,13 @@
-import { registerBehavior, swap } from "./index";
+import { Behavior, fireEvents, registerBehavior } from "./index";
 
-class FormBehavior {
+class FormBehavior extends Behavior {
   constructor(element) {
-    this.element = element;
-    element.addEventListener("submit", this.onFormSubmit.bind(this));
+    super(element);
+
+    element.addEventListener("submit", this.onSubmit.bind(this));
   }
 
-  async onFormSubmit(event) {
+  async onSubmit(event) {
     event.preventDefault();
 
     const url = this.element.action;
@@ -21,34 +22,43 @@ class FormBehavior {
       redirect: "manual", // ignore redirect
     });
 
-    // Dispatch refresh events
-    const refreshEvents = this.element.getAttribute("refresh-events");
-    if (refreshEvents) {
-      refreshEvents.split(",").forEach((eventName) => {
-        document.dispatchEvent(new CustomEvent(eventName));
-      });
+    const events = this.element.getAttribute("ld-fire");
+    if (fireEvents) {
+      fireEvents(events);
     }
-
-    // Refresh form
-    await this.refresh();
-  }
-
-  async refresh() {
-    const refreshUrl = this.element.getAttribute("refresh-url");
-    const html = await fetch(refreshUrl).then((response) => response.text());
-    swap(this.element, html);
   }
 }
 
-class FormAutoSubmitBehavior {
+class AutoSubmitBehavior extends Behavior {
   constructor(element) {
-    this.element = element;
-    this.element.addEventListener("change", () => {
-      const form = this.element.closest("form");
+    super(element);
+
+    element.addEventListener("change", () => {
+      const form = element.closest("form");
       form.dispatchEvent(new Event("submit", { cancelable: true }));
     });
   }
 }
 
+class UploadButton extends Behavior {
+  constructor(element) {
+    super(element);
+
+    const fileInput = element.nextElementSibling;
+
+    element.addEventListener("click", () => {
+      fileInput.click();
+    });
+
+    fileInput.addEventListener("change", () => {
+      const form = fileInput.closest("form");
+      const event = new Event("submit", { cancelable: true });
+      event.submitter = element;
+      form.dispatchEvent(event);
+    });
+  }
+}
+
 registerBehavior("ld-form", FormBehavior);
-registerBehavior("ld-form-auto-submit", FormAutoSubmitBehavior);
+registerBehavior("ld-auto-submit", AutoSubmitBehavior);
+registerBehavior("ld-upload-button", UploadButton);

bookmarks/frontend/behaviors/global-shortcuts.js

@@ -1,7 +1,9 @@
-import { registerBehavior } from "./index";
+import { Behavior, registerBehavior } from "./index";
+
+class GlobalShortcuts extends Behavior {
+  constructor(element) {
+    super(element);
 
-class GlobalShortcuts {
-  constructor() {
     document.addEventListener("keydown", this.onKeyDown.bind(this));
   }
 

bookmarks/frontend/behaviors/index.js

@@ -1,4 +1,35 @@
 const behaviorRegistry = {};
+const debug = false;
+
+const mutationObserver = new MutationObserver((mutations) => {
+  mutations.forEach((mutation) => {
+    mutation.removedNodes.forEach((node) => {
+      if (node instanceof HTMLElement && !node.isConnected) {
+        destroyBehaviors(node);
+      }
+    });
+    mutation.addedNodes.forEach((node) => {
+      if (node instanceof HTMLElement && node.isConnected) {
+        applyBehaviors(node);
+      }
+    });
+  });
+});
+
+mutationObserver.observe(document.body, {
+  childList: true,
+  subtree: true,
+});
+
+export class Behavior {
+  constructor(element) {
+    this.element = element;
+  }
+
+  destroy() {}
+}
+
+Behavior.interacting = false;
 
 export function registerBehavior(name, behavior) {
   behaviorRegistry[name] = behavior;
@@ -33,18 +64,82 @@ export function applyBehaviors(container, behaviorNames = null) {
 
       const behaviorInstance = new behavior(element);
       element.__behaviors.push(behaviorInstance);
+      if (debug) {
+        console.log(
+          `[Behavior] ${behaviorInstance.constructor.name} initialized`,
+        );
+      }
     });
   });
 }
 
-export function swap(element, html) {
-  const dom = new DOMParser().parseFromString(html, "text/html");
-  const newElement = dom.body.firstChild;
-  element.replaceWith(newElement);
-  applyBehaviors(newElement);
+export function destroyBehaviors(element) {
+  const behaviorNames = Object.keys(behaviorRegistry);
+
+  behaviorNames.forEach((behaviorName) => {
+    const elements = Array.from(element.querySelectorAll(`[${behaviorName}]`));
+    elements.push(element);
+
+    elements.forEach((element) => {
+      if (!element.__behaviors) {
+        return;
+      }
+
+      element.__behaviors.forEach((behavior) => {
+        behavior.destroy();
+        if (debug) {
+          console.log(`[Behavior] ${behavior.constructor.name} destroyed`);
+        }
+      });
+      delete element.__behaviors;
+    });
+  });
 }
 
-export function swapContent(element, html) {
-  element.innerHTML = html;
-  applyBehaviors(element);
+export function swap(element, html, options) {
+  const dom = new DOMParser().parseFromString(html, "text/html");
+
+  let targetElement = element;
+  let strategy = "innerHTML";
+  if (options.target) {
+    const parts = options.target.split("|");
+    targetElement =
+      parts[0] === "self" ? element : document.querySelector(parts[0]);
+    strategy = parts[1] || "innerHTML";
+  }
+
+  let contents = Array.from(dom.body.children);
+  if (options.select) {
+    contents = Array.from(dom.querySelectorAll(options.select));
+  }
+
+  switch (strategy) {
+    case "append":
+      targetElement.append(...contents);
+      break;
+    case "outerHTML":
+      targetElement.parentElement.replaceChild(contents[0], targetElement);
+      break;
+    case "innerHTML":
+    default:
+      Array.from(targetElement.children).forEach((child) => {
+        child.remove();
+      });
+      targetElement.append(...contents);
+  }
+}
+
+export function fireEvents(events) {
+  if (!events) {
+    return;
+  }
+  events.split(",").forEach((eventName) => {
+    const targets = Array.from(
+      document.querySelectorAll(`[ld-on='${eventName}']`),
+    );
+    targets.push(document);
+    targets.forEach((target) => {
+      target.dispatchEvent(new CustomEvent(eventName));
+    });
+  });
 }

bookmarks/frontend/behaviors/modal.js

@@ -1,97 +1,48 @@
-import { applyBehaviors, registerBehavior } from "./index";
+import { Behavior, registerBehavior } from "./index";
 
-class ModalBehavior {
+class ModalBehavior extends Behavior {
   constructor(element) {
-    const toggle = element;
-    toggle.addEventListener("click", this.onToggleClick.bind(this));
-    this.toggle = toggle;
+    super(element);
+
+    this.onClose = this.onClose.bind(this);
+    this.onKeyDown = this.onKeyDown.bind(this);
+
+    const modalOverlay = element.querySelector(".modal-overlay");
+    const closeButton = element.querySelector("button.close");
+    modalOverlay.addEventListener("click", this.onClose);
+    closeButton.addEventListener("click", this.onClose);
+
+    document.addEventListener("keydown", this.onKeyDown);
   }
 
-  async onToggleClick(event) {
-    // Ignore Ctrl + click
-    if (event.ctrlKey || event.metaKey) {
-      return;
-    }
-    event.preventDefault();
-    event.stopPropagation();
+  destroy() {
+    document.removeEventListener("keydown", this.onKeyDown);
+  }
 
-    // Create modal either by teleporting existing content or fetching from URL
-    const modal = this.toggle.hasAttribute("modal-content")
-      ? this.createFromContent()
-      : await this.createFromUrl();
+  onKeyDown(event) {
+    // Skip if event occurred within an input element
+    const targetNodeName = event.target.nodeName;
+    const isInputTarget =
+      targetNodeName === "INPUT" ||
+      targetNodeName === "SELECT" ||
+      targetNodeName === "TEXTAREA";
 
-    if (!modal) {
+    if (isInputTarget) {
       return;
     }
 
-    // Register close handlers
-    const modalOverlay = modal.querySelector(".modal-overlay");
-    const closeButton = modal.querySelector("button.close");
-    modalOverlay.addEventListener("click", this.onClose.bind(this));
-    closeButton.addEventListener("click", this.onClose.bind(this));
-
-    document.body.append(modal);
-    applyBehaviors(document.body);
-    this.modal = modal;
-  }
-
-  async createFromUrl() {
-    const url = this.toggle.getAttribute("modal-url");
-    const modalHtml = await fetch(url).then((response) => response.text());
-    const parser = new DOMParser();
-    const doc = parser.parseFromString(modalHtml, "text/html");
-    return doc.querySelector(".modal");
-  }
-
-  createFromContent() {
-    const contentSelector = this.toggle.getAttribute("modal-content");
-    const content = document.querySelector(contentSelector);
-    if (!content) {
-      return;
+    if (event.key === "Escape") {
+      event.preventDefault();
+      this.onClose();
     }
-
-    // Todo: make title configurable, only used for tag cloud for now
-    const modal = document.createElement("div");
-    modal.classList.add("modal", "active");
-    modal.innerHTML = `
-      <div class="modal-overlay" aria-label="Close"></div>
-      <div class="modal-container">
-        <div class="modal-header d-flex justify-between align-center">
-          <div class="modal-title h5">Tags</div>
-          <button class="close">
-          <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
-            <path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
-            <path d="M18 6l-12 12"></path>
-            <path d="M6 6l12 12"></path>
-          </svg>
-          </button>
-        </div>
-        <div class="modal-body">
-          <div class="content"></div>
-        </div>
-      </div>
-    `;
-
-    const contentOwner = content.parentElement;
-    const contentContainer = modal.querySelector(".content");
-    contentContainer.append(content);
-    this.content = content;
-    this.contentOwner = contentOwner;
-
-    return modal;
   }
 
   onClose() {
-    // Teleport content back
-    if (this.content && this.contentOwner) {
-      this.contentOwner.append(this.content);
-    }
-
-    // Remove modal
-    this.modal.classList.add("closing");
-    this.modal.addEventListener("animationend", (event) => {
+    document.removeEventListener("keydown", this.onKeyDown);
+    this.element.classList.add("closing");
+    this.element.addEventListener("animationend", (event) => {
       if (event.animationName === "fade-out") {
-        this.modal.remove();
+        this.element.remove();
       }
     });
   }

bookmarks/frontend/behaviors/tag-autocomplete.js

@@ -1,9 +1,10 @@
-import { registerBehavior } from "./index";
+import { Behavior, registerBehavior } from "./index";
 import TagAutoCompleteComponent from "../components/TagAutocomplete.svelte";
 import { ApiClient } from "../api";
 
-class TagAutocomplete {
+class TagAutocomplete extends Behavior {
   constructor(element) {
+    super(element);
     const wrapper = document.createElement("div");
     const apiBaseUrl = document.documentElement.dataset.apiBaseUrl || "";
     const apiClient = new ApiClient(apiBaseUrl);

bookmarks/frontend/index.js

@@ -2,6 +2,7 @@ import "./behaviors/bookmark-page";
 import "./behaviors/bulk-edit";
 import "./behaviors/confirm-button";
 import "./behaviors/dropdown";
+import "./behaviors/fetch";
 import "./behaviors/form";
 import "./behaviors/modal";
 import "./behaviors/global-shortcuts";

bookmarks/management/commands/full_backup.py

@@ -48,6 +48,19 @@ class Command(BaseCommand):
                         file_path = os.path.join(root, file)
                         zip_file.write(file_path, os.path.join("favicons", file))
 
+            # Backup the previews folder
+            if not os.path.exists(os.path.join("data", "previews")):
+                self.stdout.write(
+                    self.style.WARNING("No previews folder found. Skipping...")
+                )
+            else:
+                self.stdout.write("Backup bookmark previews...")
+                previews_folder = os.path.join("data", "previews")
+                for root, _, files in os.walk(previews_folder):
+                    for file in files:
+                        file_path = os.path.join(root, file)
+                        zip_file.write(file_path, os.path.join("previews", file))
+
         self.stdout.write(self.style.SUCCESS(f"Backup created at {backup_file}"))
 
     def backup_database(self, backup_db_file):

bookmarks/migrations/0033_userprofile_default_mark_unread.py

@@ -0,0 +1,18 @@
+# Generated by Django 5.0.3 on 2024-04-17 19:27
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("bookmarks", "0032_html_snapshots_hint_toast"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="userprofile",
+            name="default_mark_unread",
+            field=models.BooleanField(default=False),
+        ),
+    ]

bookmarks/migrations/0034_bookmark_preview_image_file_and_more.py

@@ -0,0 +1,23 @@
+# Generated by Django 5.0.3 on 2024-05-10 07:01
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("bookmarks", "0033_userprofile_default_mark_unread"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="bookmark",
+            name="preview_image_file",
+            field=models.CharField(blank=True, max_length=512),
+        ),
+        migrations.AddField(
+            model_name="userprofile",
+            name="enable_preview_images",
+            field=models.BooleanField(default=False),
+        ),
+    ]

bookmarks/migrations/0035_userprofile_tag_grouping.py

@@ -0,0 +1,22 @@
+# Generated by Django 5.0.3 on 2024-05-14 08:28
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("bookmarks", "0034_bookmark_preview_image_file_and_more"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="userprofile",
+            name="tag_grouping",
+            field=models.CharField(
+                choices=[("alphabetical", "Alphabetical"), ("disabled", "Disabled")],
+                default="alphabetical",
+                max_length=12,
+            ),
+        ),
+    ]

bookmarks/migrations/0036_userprofile_auto_tagging_rules.py

@@ -0,0 +1,18 @@
+# Generated by Django 5.0.3 on 2024-05-17 07:09
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("bookmarks", "0035_userprofile_tag_grouping"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="userprofile",
+            name="auto_tagging_rules",
+            field=models.TextField(blank=True),
+        ),
+    ]

bookmarks/models.py

@@ -59,6 +59,7 @@ class Bookmark(models.Model):
     website_description = models.TextField(blank=True, null=True)
     web_archive_snapshot_url = models.CharField(max_length=2048, blank=True)
     favicon_file = models.CharField(max_length=512, blank=True)
+    preview_image_file = models.CharField(max_length=512, blank=True)
     unread = models.BooleanField(default=False)
     is_archived = models.BooleanField(default=False)
     shared = models.BooleanField(default=False)
@@ -91,6 +92,7 @@ class Bookmark(models.Model):
 
 class BookmarkAsset(models.Model):
     TYPE_SNAPSHOT = "snapshot"
+    TYPE_UPLOAD = "upload"
 
     CONTENT_TYPE_HTML = "text/html"
 
@@ -118,6 +120,9 @@ class BookmarkAsset(models.Model):
                 pass
         super().save(*args, **kwargs)
 
+    def __str__(self):
+        return self.display_name or f"Bookmark Asset #{self.pk}"
+
 
 @receiver(post_delete, sender=BookmarkAsset)
 def bookmark_asset_deleted(sender, instance, **kwargs):
@@ -347,6 +352,12 @@ class UserProfile(models.Model):
         (TAG_SEARCH_STRICT, "Strict"),
         (TAG_SEARCH_LAX, "Lax"),
     ]
+    TAG_GROUPING_ALPHABETICAL = "alphabetical"
+    TAG_GROUPING_DISABLED = "disabled"
+    TAG_GROUPING_CHOICES = [
+        (TAG_GROUPING_ALPHABETICAL, "Alphabetical"),
+        (TAG_GROUPING_DISABLED, "Disabled"),
+    ]
     user = models.OneToOneField(
         get_user_model(), related_name="profile", on_delete=models.CASCADE
     )
@@ -387,9 +398,16 @@ class UserProfile(models.Model):
         blank=False,
         default=TAG_SEARCH_STRICT,
     )
+    tag_grouping = models.CharField(
+        max_length=12,
+        choices=TAG_GROUPING_CHOICES,
+        blank=False,
+        default=TAG_GROUPING_ALPHABETICAL,
+    )
     enable_sharing = models.BooleanField(default=False, null=False)
     enable_public_sharing = models.BooleanField(default=False, null=False)
     enable_favicons = models.BooleanField(default=False, null=False)
+    enable_preview_images = models.BooleanField(default=False, null=False)
     display_url = models.BooleanField(default=False, null=False)
     display_view_bookmark_action = models.BooleanField(default=True, null=False)
     display_edit_bookmark_action = models.BooleanField(default=True, null=False)
@@ -397,8 +415,10 @@ class UserProfile(models.Model):
     display_remove_bookmark_action = models.BooleanField(default=True, null=False)
     permanent_notes = models.BooleanField(default=False, null=False)
     custom_css = models.TextField(blank=True, null=False)
+    auto_tagging_rules = models.TextField(blank=True, null=False)
     search_preferences = models.JSONField(default=dict, null=False)
     enable_automatic_html_snapshots = models.BooleanField(default=True, null=False)
+    default_mark_unread = models.BooleanField(default=False, null=False)
 
 
 class UserProfileForm(forms.ModelForm):
@@ -412,9 +432,11 @@ class UserProfileForm(forms.ModelForm):
             "bookmark_link_target",
             "web_archive_integration",
             "tag_search",
+            "tag_grouping",
             "enable_sharing",
             "enable_public_sharing",
             "enable_favicons",
+            "enable_preview_images",
             "enable_automatic_html_snapshots",
             "display_url",
             "display_view_bookmark_action",
@@ -422,7 +444,9 @@ class UserProfileForm(forms.ModelForm):
             "display_archive_bookmark_action",
             "display_remove_bookmark_action",
             "permanent_notes",
+            "default_mark_unread",
             "custom_css",
+            "auto_tagging_rules",
         ]
 
 

bookmarks/services/auto_tagging.py

@@ -0,0 +1,70 @@
+from urllib.parse import urlparse, parse_qs
+import re
+import idna
+
+
+def get_tags(script: str, url: str):
+    parsed_url = urlparse(url.lower())
+    result = set()
+
+    for line in script.lower().split("\n"):
+        if "#" in line:
+            i = line.index("#")
+            line = line[:i]
+
+        parts = line.split()
+        if len(parts) < 2:
+            continue
+
+        domain_pattern = re.sub("^https?://", "", parts[0])
+        path_pattern = None
+        qs_pattern = None
+
+        if "/" in domain_pattern:
+            i = domain_pattern.index("/")
+            path_pattern = domain_pattern[i:]
+            domain_pattern = domain_pattern[:i]
+
+        if path_pattern and "?" in path_pattern:
+            i = path_pattern.index("?")
+            qs_pattern = path_pattern[i + 1 :]
+            path_pattern = path_pattern[:i]
+
+        if not _domains_matches(domain_pattern, parsed_url.netloc):
+            continue
+
+        if path_pattern and not _path_matches(path_pattern, parsed_url.path):
+            continue
+
+        if qs_pattern and not _qs_matches(qs_pattern, parsed_url.query):
+            continue
+
+        for tag in parts[1:]:
+            result.add(tag)
+
+    return result
+
+
+def _path_matches(expected_path: str, actual_path: str) -> bool:
+    return actual_path.startswith(expected_path)
+
+
+def _domains_matches(expected_domain: str, actual_domain: str) -> bool:
+    expected_domain = idna.encode(expected_domain)
+    actual_domain = idna.encode(actual_domain)
+
+    return actual_domain.endswith(expected_domain)
+
+
+def _qs_matches(expected_qs: str, actual_qs: str) -> bool:
+    expected_qs = parse_qs(expected_qs, keep_blank_values=True)
+    actual_qs = parse_qs(actual_qs, keep_blank_values=True)
+
+    for key in expected_qs:
+        if key not in actual_qs:
+            return False
+        for value in expected_qs[key]:
+            if value != "" and value not in actual_qs[key]:
+                return False
+
+    return True

bookmarks/services/bookmarks.py

@@ -1,12 +1,19 @@
+import logging
+import os
 from typing import Union
 
+from django.conf import settings
 from django.contrib.auth.models import User
+from django.core.files.uploadedfile import UploadedFile
 from django.utils import timezone
 
-from bookmarks.models import Bookmark, parse_tag_string
-from bookmarks.services.tags import get_or_create_tags
-from bookmarks.services import website_loader
+from bookmarks.models import Bookmark, BookmarkAsset, parse_tag_string
 from bookmarks.services import tasks
+from bookmarks.services import website_loader
+from bookmarks.services import auto_tagging
+from bookmarks.services.tags import get_or_create_tags
+
+logger = logging.getLogger(__name__)
 
 
 def create_bookmark(bookmark: Bookmark, tag_string: str, current_user: User):
@@ -34,6 +41,8 @@ def create_bookmark(bookmark: Bookmark, tag_string: str, current_user: User):
     tasks.create_web_archive_snapshot(current_user, bookmark, False)
     # Load favicon
     tasks.load_favicon(current_user, bookmark)
+    # Load preview image
+    tasks.load_preview_image(current_user, bookmark)
     # Create HTML snapshot
     if current_user.profile.enable_automatic_html_snapshots:
         tasks.create_html_snapshot(bookmark)
@@ -52,6 +61,8 @@ def update_bookmark(bookmark: Bookmark, tag_string, current_user: User):
     bookmark.save()
     # Update favicon
     tasks.load_favicon(current_user, bookmark)
+    # Update preview image
+    tasks.load_preview_image(current_user, bookmark)
 
     if has_url_changed:
         # Update web archive snapshot, if URL changed
@@ -176,6 +187,46 @@ def unshare_bookmarks(bookmark_ids: [Union[int, str]], current_user: User):
     )
 
 
+def _generate_upload_asset_filename(asset: BookmarkAsset, filename: str):
+    formatted_datetime = asset.date_created.strftime("%Y-%m-%d_%H%M%S")
+    return f"{asset.asset_type}_{formatted_datetime}_{filename}"
+
+
+def upload_asset(bookmark: Bookmark, upload_file: UploadedFile) -> BookmarkAsset:
+    asset = BookmarkAsset(
+        bookmark=bookmark,
+        asset_type=BookmarkAsset.TYPE_UPLOAD,
+        content_type=upload_file.content_type,
+        display_name=upload_file.name,
+        status=BookmarkAsset.STATUS_PENDING,
+        gzip=False,
+    )
+    asset.save()
+
+    try:
+        filename = _generate_upload_asset_filename(asset, upload_file.name)
+        filepath = os.path.join(settings.LD_ASSET_FOLDER, filename)
+        with open(filepath, "wb") as f:
+            for chunk in upload_file.chunks():
+                f.write(chunk)
+        asset.status = BookmarkAsset.STATUS_COMPLETE
+        asset.file = filename
+        asset.file_size = upload_file.size
+        logger.info(
+            f"Successfully uploaded asset file. bookmark={bookmark} file={upload_file.name}"
+        )
+    except Exception as e:
+        logger.error(
+            f"Failed to upload asset file. bookmark={bookmark} file={upload_file.name}",
+            exc_info=e,
+        )
+        asset.status = BookmarkAsset.STATUS_FAILURE
+
+    asset.save()
+
+    return asset
+
+
 def _merge_bookmark_data(from_bookmark: Bookmark, to_bookmark: Bookmark):
     to_bookmark.title = from_bookmark.title
     to_bookmark.description = from_bookmark.description
@@ -192,6 +243,15 @@ def _update_website_metadata(bookmark: Bookmark):
 
 def _update_bookmark_tags(bookmark: Bookmark, tag_string: str, user: User):
     tag_names = parse_tag_string(tag_string)
+
+    if user.profile.auto_tagging_rules:
+        auto_tag_names = auto_tagging.get_tags(
+            user.profile.auto_tagging_rules, bookmark.url
+        )
+        for auto_tag_name in auto_tag_names:
+            if auto_tag_name not in tag_names:
+                tag_names.append(auto_tag_name)
+
     tags = get_or_create_tags(tag_names, user)
     bookmark.tags.set(tags)
 

bookmarks/services/importer.py

@@ -79,10 +79,10 @@ def import_netscape_html(
     for batch in batches:
         _import_batch(batch, user, options, tag_cache, result)
 
-    # Create snapshots for newly imported bookmarks
-    tasks.schedule_bookmarks_without_snapshots(user)
     # Load favicons for newly imported bookmarks
     tasks.schedule_bookmarks_without_favicons(user)
+    # Load previews for newly imported bookmarks
+    tasks.schedule_bookmarks_without_previews(user)
 
     end = timezone.now()
     logger.debug(f"Import duration: {end - import_start}")

bookmarks/services/preview_image_loader.py

@@ -0,0 +1,88 @@
+import logging
+import mimetypes
+import os.path
+import hashlib
+from pathlib import Path
+
+import requests
+from django.conf import settings
+from bookmarks.services import website_loader
+
+logger = logging.getLogger(__name__)
+
+
+def _ensure_preview_folder():
+    Path(settings.LD_PREVIEW_FOLDER).mkdir(parents=True, exist_ok=True)
+
+
+def _url_to_filename(preview_image: str) -> str:
+    return hashlib.md5(preview_image.encode()).hexdigest()
+
+
+def _get_image_path(preview_image_file: str) -> Path:
+    return Path(os.path.join(settings.LD_PREVIEW_FOLDER, preview_image_file))
+
+
+def load_preview_image(url: str) -> str | None:
+    _ensure_preview_folder()
+
+    metadata = website_loader.load_website_metadata(url)
+    if not metadata.preview_image:
+        logger.debug(f"Could not find preview image in metadata: {url}")
+        return None
+
+    image_url = metadata.preview_image
+
+    logger.debug(f"Loading preview image: {image_url}")
+    with requests.get(image_url, stream=True) as response:
+        if response.status_code < 200 or response.status_code >= 300:
+            logger.debug(
+                f"Bad response status code for preview image: {image_url} status_code={response.status_code}"
+            )
+            return None
+
+        if "Content-Length" not in response.headers:
+            logger.debug(f"Empty Content-Length for preview image: {image_url}")
+            return None
+
+        content_length = int(response.headers["Content-Length"])
+        if content_length > settings.LD_PREVIEW_MAX_SIZE:
+            logger.debug(
+                f"Content-Length exceeds LD_PREVIEW_MAX_SIZE: {image_url} length={content_length}"
+            )
+            return None
+
+        if "Content-Type" not in response.headers:
+            logger.debug(f"Empty Content-Type for preview image: {image_url}")
+            return None
+
+        content_type = response.headers["Content-Type"].split(";", 1)[0]
+        file_extension = mimetypes.guess_extension(content_type)
+
+        if file_extension not in settings.LD_PREVIEW_ALLOWED_EXTENSIONS:
+            logger.debug(
+                f"Unsupported Content-Type for preview image: {image_url} content_type={content_type}"
+            )
+            return None
+
+        preview_image_hash = _url_to_filename(url)
+        preview_image_file = f"{preview_image_hash}{file_extension}"
+        preview_image_path = _get_image_path(preview_image_file)
+
+        with open(preview_image_path, "wb") as file:
+            downloaded = 0
+            for chunk in response.iter_content(chunk_size=8192):
+                downloaded += len(chunk)
+                if downloaded > content_length:
+                    logger.debug(
+                        f"Content-Length mismatch for preview image: {image_url} length={content_length} downloaded={downloaded}"
+                    )
+                    file.close()
+                    preview_image_path.unlink()
+                    return None
+
+                file.write(chunk)
+
+    logger.debug(f"Saved preview image as: {preview_image_path}")
+
+    return preview_image_file

bookmarks/services/singlefile.py

@@ -18,11 +18,12 @@ logger = logging.getLogger(__name__)
 
 def create_snapshot(url: str, filepath: str):
     singlefile_path = settings.LD_SINGLEFILE_PATH
-    # parse string to list of arguments
-    singlefile_options = shlex.split(settings.LD_SINGLEFILE_OPTIONS)
+    # parse options to list of arguments
+    ublock_options = shlex.split(settings.LD_SINGLEFILE_UBLOCK_OPTIONS)
+    custom_options = shlex.split(settings.LD_SINGLEFILE_OPTIONS)
     temp_filepath = filepath + ".tmp"
     # concat lists
-    args = [singlefile_path] + singlefile_options + [url, temp_filepath]
+    args = [singlefile_path] + ublock_options + custom_options + [url, temp_filepath]
     try:
         # Use start_new_session=True to create a new process group
         process = subprocess.Popen(args, start_new_session=True)

bookmarks/services/tasks.py

@@ -1,20 +1,21 @@
 import functools
 import logging
 import os
+from typing import List
 
 import waybackpy
 from django.conf import settings
 from django.contrib.auth import get_user_model
 from django.contrib.auth.models import User
+from django.db.models import Q
 from django.utils import timezone, formats
 from huey import crontab
 from huey.contrib.djhuey import HUEY as huey
 from huey.exceptions import TaskLockedException
-from waybackpy.exceptions import WaybackError, TooManyRequestsError, NoCDXRecordFound
+from waybackpy.exceptions import WaybackError, TooManyRequestsError
 
-import bookmarks.services.wayback
 from bookmarks.models import Bookmark, BookmarkAsset, UserProfile
-from bookmarks.services import favicon_loader, singlefile
+from bookmarks.services import favicon_loader, singlefile, preview_image_loader
 from bookmarks.services.website_loader import DEFAULT_USER_AGENT
 
 logger = logging.getLogger(__name__)
@@ -64,29 +65,6 @@ def create_web_archive_snapshot(user: User, bookmark: Bookmark, force_update: bo
         _create_web_archive_snapshot_task(bookmark.id, force_update)
 
 
-def _load_newest_snapshot(bookmark: Bookmark):
-    try:
-        logger.info(f"Load existing snapshot for bookmark. url={bookmark.url}")
-        cdx_api = bookmarks.services.wayback.CustomWaybackMachineCDXServerAPI(
-            bookmark.url
-        )
-        existing_snapshot = cdx_api.newest()
-
-        if existing_snapshot:
-            bookmark.web_archive_snapshot_url = existing_snapshot.archive_url
-            bookmark.save(update_fields=["web_archive_snapshot_url"])
-            logger.info(
-                f"Using newest snapshot. url={bookmark.url} from={existing_snapshot.datetime_timestamp}"
-            )
-
-    except NoCDXRecordFound:
-        logger.info(f"Could not find any snapshots for bookmark. url={bookmark.url}")
-    except WaybackError as error:
-        logger.error(
-            f"Failed to load existing snapshot. url={bookmark.url}", exc_info=error
-        )
-
-
 def _create_snapshot(bookmark: Bookmark):
     logger.info(f"Create new snapshot for bookmark. url={bookmark.url}...")
     archive = waybackpy.WaybackMachineSaveAPI(
@@ -115,48 +93,27 @@ def _create_web_archive_snapshot_task(bookmark_id: int, force_update: bool):
         return
     except TooManyRequestsError:
         logger.error(
-            f"Failed to create snapshot due to rate limiting, trying to load newest snapshot as fallback. url={bookmark.url}"
+            f"Failed to create snapshot due to rate limiting. url={bookmark.url}"
         )
     except WaybackError as error:
         logger.error(
-            f"Failed to create snapshot, trying to load newest snapshot as fallback. url={bookmark.url}",
+            f"Failed to create snapshot. url={bookmark.url}",
             exc_info=error,
         )
 
-    # Load the newest snapshot as fallback
-    _load_newest_snapshot(bookmark)
-
 
 @task()
 def _load_web_archive_snapshot_task(bookmark_id: int):
-    try:
-        bookmark = Bookmark.objects.get(id=bookmark_id)
-    except Bookmark.DoesNotExist:
-        return
-    # Skip if snapshot exists
-    if bookmark.web_archive_snapshot_url:
-        return
-    # Load the newest snapshot
-    _load_newest_snapshot(bookmark)
-
-
-def schedule_bookmarks_without_snapshots(user: User):
-    if is_web_archive_integration_active(user):
-        _schedule_bookmarks_without_snapshots_task(user.id)
+    # Loading snapshots from CDX API has been removed, keeping the task function
+    # for now to prevent errors when huey tries to run the task
+    pass
 
 
 @task()
 def _schedule_bookmarks_without_snapshots_task(user_id: int):
-    user = get_user_model().objects.get(id=user_id)
-    bookmarks_without_snapshots = Bookmark.objects.filter(
-        web_archive_snapshot_url__exact="", owner=user
-    )
-
-    # TODO: Implement bulk task creation
-    for bookmark in bookmarks_without_snapshots:
-        # To prevent rate limit errors from the Wayback API only try to load the latest snapshots instead of creating
-        # new ones when processing bookmarks in bulk
-        _load_web_archive_snapshot_task(bookmark.id)
+    # Loading snapshots from CDX API has been removed, keeping the task function
+    # for now to prevent errors when huey tries to run the task
+    pass
 
 
 def is_favicon_feature_active(user: User) -> bool:
@@ -165,6 +122,12 @@ def is_favicon_feature_active(user: User) -> bool:
     return background_tasks_enabled and user.profile.enable_favicons
 
 
+def is_preview_feature_active(user: User) -> bool:
+    return (
+        user.profile.enable_preview_images and not settings.LD_DISABLE_BACKGROUND_TASKS
+    )
+
+
 def load_favicon(user: User, bookmark: Bookmark):
     if is_favicon_feature_active(user):
         _load_favicon_task(bookmark.id)
@@ -220,6 +183,51 @@ def _schedule_refresh_favicons_task(user_id: int):
         _load_favicon_task(bookmark.id)
 
 
+def load_preview_image(user: User, bookmark: Bookmark):
+    if is_preview_feature_active(user):
+        _load_preview_image_task(bookmark.id)
+
+
+@task()
+def _load_preview_image_task(bookmark_id: int):
+    try:
+        bookmark = Bookmark.objects.get(id=bookmark_id)
+    except Bookmark.DoesNotExist:
+        return
+
+    logger.info(f"Load preview image for bookmark. url={bookmark.url}")
+
+    new_preview_image_file = preview_image_loader.load_preview_image(bookmark.url)
+
+    if new_preview_image_file != bookmark.preview_image_file:
+        bookmark.preview_image_file = new_preview_image_file or ""
+        bookmark.save(update_fields=["preview_image_file"])
+        logger.info(
+            f"Successfully updated preview image for bookmark. url={bookmark.url} preview_image_file={new_preview_image_file}"
+        )
+
+
+def schedule_bookmarks_without_previews(user: User):
+    if is_preview_feature_active(user):
+        _schedule_bookmarks_without_previews_task(user.id)
+
+
+@task()
+def _schedule_bookmarks_without_previews_task(user_id: int):
+    user = get_user_model().objects.get(id=user_id)
+    bookmarks = Bookmark.objects.filter(
+        Q(preview_image_file__exact=""),
+        owner=user,
+    )
+
+    # TODO: Implement bulk task creation
+    for bookmark in bookmarks:
+        try:
+            _load_preview_image_task(bookmark.id)
+        except Exception as exc:
+            logging.exception(exc)
+
+
 def is_html_snapshot_feature_active() -> bool:
     return settings.LD_ENABLE_SNAPSHOTS and not settings.LD_DISABLE_BACKGROUND_TASKS
 
@@ -228,6 +236,26 @@ def create_html_snapshot(bookmark: Bookmark):
     if not is_html_snapshot_feature_active():
         return
 
+    asset = _create_snapshot_asset(bookmark)
+    asset.save()
+
+
+def create_html_snapshots(bookmark_list: List[Bookmark]):
+    if not is_html_snapshot_feature_active():
+        return
+
+    assets_to_create = []
+    for bookmark in bookmark_list:
+        asset = _create_snapshot_asset(bookmark)
+        assets_to_create.append(asset)
+
+    BookmarkAsset.objects.bulk_create(assets_to_create)
+
+
+MAX_SNAPSHOT_FILENAME_LENGTH = 192
+
+
+def _create_snapshot_asset(bookmark: Bookmark) -> BookmarkAsset:
     timestamp = formats.date_format(timezone.now(), "SHORT_DATE_FORMAT")
     asset = BookmarkAsset(
         bookmark=bookmark,
@@ -236,10 +264,7 @@ def create_html_snapshot(bookmark: Bookmark):
         display_name=f"HTML snapshot from {timestamp}",
         status=BookmarkAsset.STATUS_PENDING,
     )
-    asset.save()
-
-
-MAX_SNAPSHOT_FILENAME_LENGTH = 192
+    return asset
 
 
 def _generate_snapshot_filename(asset: BookmarkAsset) -> str:
@@ -305,3 +330,23 @@ def _create_html_snapshot_task(asset_id: int):
         )
         asset.status = BookmarkAsset.STATUS_FAILURE
         asset.save()
+
+
+def create_missing_html_snapshots(user: User) -> int:
+    if not is_html_snapshot_feature_active():
+        return 0
+
+    bookmarks_without_snapshots = Bookmark.objects.filter(owner=user).exclude(
+        bookmarkasset__asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+        bookmarkasset__status__in=[
+            BookmarkAsset.STATUS_PENDING,
+            BookmarkAsset.STATUS_COMPLETE,
+        ],
+    )
+    bookmarks_without_snapshots |= Bookmark.objects.filter(owner=user).exclude(
+        bookmarkasset__asset_type=BookmarkAsset.TYPE_SNAPSHOT
+    )
+
+    create_html_snapshots(list(bookmarks_without_snapshots))
+
+    return bookmarks_without_snapshots.count()

bookmarks/services/wayback.py

@@ -1,42 +1,20 @@
-import time
-from typing import Dict
+import datetime
 
-import waybackpy
-import waybackpy.utils
-from waybackpy.exceptions import NoCDXRecordFound
+from django.utils import timezone
 
 
-class CustomWaybackMachineCDXServerAPI(waybackpy.WaybackMachineCDXServerAPI):
+def generate_fallback_webarchive_url(
+    url: str, timestamp: datetime.datetime
+) -> str | None:
     """
-    Customized WaybackMachineCDXServerAPI to work around some issues with retrieving the newest snapshot.
-    See https://github.com/akamhy/waybackpy/issues/176
+    Generate a URL to the web archive for the given URL and timestamp.
+    A snapshot for the specific timestamp might not exist, in which case the
+    web archive will show the closest snapshot to the given timestamp.
+    If there is no snapshot at all the URL will be invalid.
     """
+    if not url:
+        return None
+    if not timestamp:
+        timestamp = timezone.now()
 
-    def newest(self):
-        unix_timestamp = int(time.time())
-        self.closest = waybackpy.utils.unix_timestamp_to_wayback_timestamp(
-            unix_timestamp
-        )
-        self.sort = "closest"
-        self.limit = -5
-
-        newest_snapshot = None
-        for snapshot in self.snapshots():
-            newest_snapshot = snapshot
-            break
-
-        if not newest_snapshot:
-            raise NoCDXRecordFound(
-                "Wayback Machine's CDX server did not return any records "
-                + "for the query. The URL may not have any archives "
-                + " on the Wayback Machine or the URL may have been recently "
-                + "archived and is still not available on the CDX server."
-            )
-
-        return newest_snapshot
-
-    def add_payload(self, payload: Dict[str, str]) -> None:
-        super().add_payload(payload)
-        # Set fastLatest query param, as we are only using this API to get the latest snapshot and using fastLatest
-        # makes searching for latest snapshots faster
-        payload["fastLatest"] = "true"
+    return f"https://web.archive.org/web/{timestamp.strftime('%Y%m%d%H%M%S')}/{url}"

bookmarks/services/website_loader.py

@@ -1,6 +1,7 @@
 import logging
 from dataclasses import dataclass
 from functools import lru_cache
+from urllib.parse import urljoin
 
 import requests
 from bs4 import BeautifulSoup
@@ -15,12 +16,14 @@ class WebsiteMetadata:
     url: str
     title: str
     description: str
+    preview_image: str | None
 
     def to_dict(self):
         return {
             "url": self.url,
             "title": self.title,
             "description": self.description,
+            "preview_image": self.preview_image,
         }
 
 
@@ -30,6 +33,7 @@ class WebsiteMetadata:
 def load_website_metadata(url: str):
     title = None
     description = None
+    preview_image = None
     try:
         start = timezone.now()
         page_text = load_page(url)
@@ -55,10 +59,21 @@ def load_website_metadata(url: str):
                 else None
             )
 
+        image_tag = soup.find("meta", attrs={"property": "og:image"})
+        preview_image = image_tag["content"].strip() if image_tag else None
+        if (
+            preview_image
+            and not preview_image.startswith("http://")
+            and not preview_image.startswith("https://")
+        ):
+            preview_image = urljoin(url, preview_image)
+
         end = timezone.now()
         logger.debug(f"Parsing duration: {end - start}")
     finally:
-        return WebsiteMetadata(url=url, title=title, description=description)
+        return WebsiteMetadata(
+            url=url, title=title, description=description, preview_image=preview_image
+        )
 
 
 CHUNK_SIZE = 50 * 1024

bookmarks/signals.py

@@ -1,15 +1,7 @@
 from django.conf import settings
-from django.contrib.auth import user_logged_in
 from django.db.backends.signals import connection_created
 from django.dispatch import receiver
 
-from bookmarks.services import tasks
-
-
-@receiver(user_logged_in)
-def user_logged_in(sender, request, user, **kwargs):
-    tasks.schedule_bookmarks_without_snapshots(user)
-
 
 @receiver(connection_created)
 def extend_sqlite(connection=None, **kwargs):

bookmarks/styles/bookmark-details.scss

@@ -33,6 +33,15 @@
     text-overflow: ellipsis;
   }
 
+  .preview-image {
+    margin: $unit-4 0;
+
+    img {
+      max-width: 100%;
+      max-height: 200px;
+    }
+  }
+
   dl {
     margin-bottom: 0;
   }
@@ -61,16 +70,22 @@
 
   .assets .asset-text {
     flex: 1 1 0;
+    gap: $unit-2;
+    min-width: 0;
+    display: flex;
+  }
+
+  .assets .asset-text .truncate {
+    flex-shrink: 1;
   }
 
   .assets .asset-text .filesize {
     color: $gray-color;
-    margin-left: $unit-2;
   }
 
   .assets .asset-actions, .assets-actions {
     display: flex;
-    gap: $unit-3;
+    gap: $unit-4;
     align-items: center;
   }
 

bookmarks/styles/bookmark-form.scss

@@ -36,12 +36,11 @@
   .form-input-hint.bookmark-exists {
     display: none;
     color: $warning-color;
+  }
 
-    a {
-      color: $warning-color;
-      text-decoration: underline;
-      font-weight: bold;
-    }
+  .form-input-hint.auto-tags {
+    display: none;
+    color: $success-color;
   }
 
   details.notes textarea {

bookmarks/styles/bookmark-page.scss

@@ -128,9 +128,26 @@ ul.bookmark-list {
 /* Bookmarks */
 li[ld-bookmark-item] {
   position: relative;
+  display: flex;
+  gap: $unit-2;
   margin-top: $unit-2;
 
-  [ld-bulk-edit-checkbox].form-checkbox {
+  .content {
+    flex: 1 1 0;
+    min-width: 0;
+  }
+
+  img.preview-image {
+    flex: 0 0 auto;
+    width: 100px;
+    height: 60px;
+    margin-top: $unit-h;
+    object-fit: cover;
+    border-radius: $border-radius;
+    border: solid 1px $border-color-dark;
+  }
+
+  .form-checkbox.bulk-edit-checkbox {
     display: none;
   }
 
@@ -182,6 +199,12 @@ li[ld-bookmark-item] {
     animation: 0.3s ease 0s appear;
   }
 
+  @media (pointer:coarse) {
+    .title a[data-tooltip]::after {
+      display: none;
+    }
+  }
+
   &.unread .title a {
     font-style: italic;
   }
@@ -323,7 +346,7 @@ $bulk-edit-transition-duration: 400ms;
   }
 
   /* All checkbox */
-  [ld-bulk-edit-checkbox][all].form-checkbox {
+  .form-checkbox.bulk-edit-checkbox.all {
     display: block;
     width: $bulk-edit-toggle-width;
     margin: 0 0 0 $bulk-edit-toggle-offset;
@@ -331,7 +354,7 @@ $bulk-edit-transition-duration: 400ms;
   }
 
   /* Bookmark checkboxes */
-  li[ld-bookmark-item] [ld-bulk-edit-checkbox].form-checkbox {
+  li[ld-bookmark-item] .form-checkbox.bulk-edit-checkbox {
     display: block;
     position: absolute;
     width: $bulk-edit-toggle-width;
@@ -346,11 +369,11 @@ $bulk-edit-transition-duration: 400ms;
     transition: all $bulk-edit-transition-duration;
 
     .form-icon {
-        top: 0;
+      top: 0;
     }
   }
 
-  &.active li[ld-bookmark-item] [ld-bulk-edit-checkbox].form-checkbox {
+  &.active li[ld-bookmark-item] .form-checkbox.bulk-edit-checkbox {
     visibility: visible;
     opacity: 1;
   }

bookmarks/styles/reader-mode.scss

@@ -0,0 +1,27 @@
+html.reader-mode {
+  --font-size: 1rem;
+  line-height: 1.6;
+
+  body {
+    margin: 3rem 2rem;
+  }
+
+  .container {
+    max-width: 600px;
+  }
+
+  .byline {
+    font-style: italic;
+    font-size: 0.8rem;
+  }
+
+  .reading-time {
+    font-size: 0.7rem;
+  }
+
+  img {
+    max-width: 100%;
+    height: auto;
+  }
+}
+

bookmarks/styles/settings.scss

@@ -7,8 +7,9 @@
     }
   }
 
-  textarea.custom-css {
+  textarea.monospace {
     font-family: monospace;
+    box-sizing: border-box;
   }
 
   .input-group > input[type=submit] {

bookmarks/styles/spectre.scss

@@ -195,3 +195,10 @@ ul.menu li:first-child {
     font-size: 16px;
   }
 }
+
+// Hide tooltips on mobile
+@media (pointer:coarse) {
+  .tooltip::after {
+    display: none;
+  }
+}

bookmarks/styles/theme-dark.scss

@@ -12,6 +12,7 @@
 @import "bookmark-form";
 @import "settings";
 @import "markdown";
+@import "reader-mode";
 
 /* Dark theme overrides */
 

bookmarks/styles/theme-light.scss

@@ -12,3 +12,4 @@
 @import "bookmark-form";
 @import "settings";
 @import "markdown";
+@import "reader-mode";

bookmarks/templates/bookmarks/archive.html

@@ -4,11 +4,7 @@
 {% load bookmarks %}
 
 {% block content %}
-  <div class="bookmarks-page grid columns-md-1"
-       ld-bulk-edit
-       ld-bookmark-page
-       bookmarks-url="{% url 'bookmarks:partials.bookmark_list.archived' %}"
-       tags-url="{% url 'bookmarks:partials.tag_cloud.archived' %}">
+  <div ld-bulk-edit class="bookmarks-page grid columns-md-1">
 
     {# Bookmark list #}
     <section class="content-area col-2">
@@ -17,17 +13,22 @@
         <div class="header-controls">
           {% bookmark_search bookmark_list.search tag_cloud.tags mode='archived' %}
           {% include 'bookmarks/bulk_edit/toggle.html' %}
-          <button ld-modal modal-content=".tag-cloud" class="btn ml-2 show-md">Tags</button>
+          <button ld-fetch="{{ bookmark_list.tag_modal_url }}" ld-target="body|append" ld-on="click"
+                  class="btn ml-2 show-md">Tags
+          </button>
         </div>
       </div>
 
-      <form class="bookmark-actions"
+      <form ld-form ld-fire="refresh-bookmark-list,refresh-tag-cloud"
+            class="bookmark-actions"
             action="{{ bookmark_list.action_url|safe }}"
             method="post" autocomplete="off">
         {% csrf_token %}
         {% include 'bookmarks/bulk_edit/bar.html' with disable_actions='bulk_archive' %}
 
-        <div class="bookmark-list-container">
+        <div ld-fetch="{{ bookmark_list.refresh_url }}" ld-on="refresh-bookmark-list"
+             ld-fire="refresh-bookmark-list-done"
+             class="bookmark-list-container">
           {% include 'bookmarks/bookmark_list.html' %}
         </div>
       </form>
@@ -38,7 +39,8 @@
       <div class="content-area-header">
         <h2>Tags</h2>
       </div>
-      <div class="tag-cloud-container">
+      <div ld-fetch="{{ tag_cloud.refresh_url }}" ld-on="refresh-tag-cloud"
+           class="tag-cloud-container">
         {% include 'bookmarks/tag_cloud.html' %}
       </div>
     </section>

bookmarks/templates/bookmarks/bookmark_list.html

@@ -10,142 +10,143 @@
       data-bookmarks-total="{{ bookmark_list.bookmarks_total }}">
     {% for bookmark_item in bookmark_list.items %}
       <li ld-bookmark-item{% if bookmark_item.css_classes %} class="{{ bookmark_item.css_classes }}"{% endif %}>
-        <div class="title">
-          <label ld-bulk-edit-checkbox class="form-checkbox">
-            <input type="checkbox" name="bookmark_id" value="{{ bookmark_item.id }}">
-            <i class="form-icon"></i>
-          </label>
-          {% if bookmark_item.favicon_file and bookmark_list.show_favicons %}
-            <img src="{% static bookmark_item.favicon_file %}" alt="">
-          {% endif %}
-          <a href="{{ bookmark_item.url }}" target="{{ bookmark_list.link_target }}" rel="noopener">
-            <span>{{ bookmark_item.title }}</span>
-          </a>
-        </div>
-        {% if bookmark_list.show_url %}
-          <div class="url-path truncate">
-            <a href="{{ bookmark_item.url }}" target="{{ bookmark_list.link_target }}" rel="noopener"
-               class="url-display">
-              {{ bookmark_item.url }}
+        <div class="content">
+          <div class="title">
+            <label class="form-checkbox bulk-edit-checkbox">
+              <input type="checkbox" name="bookmark_id" value="{{ bookmark_item.id }}">
+              <i class="form-icon"></i>
+            </label>
+            {% if bookmark_item.favicon_file and bookmark_list.show_favicons %}
+              <img class="favicon" src="{% static bookmark_item.favicon_file %}" alt="">
+            {% endif %}
+            <a href="{{ bookmark_item.url }}" target="{{ bookmark_list.link_target }}" rel="noopener">
+              <span>{{ bookmark_item.title }}</span>
             </a>
           </div>
-        {% endif %}
-        {% if bookmark_list.description_display == 'inline' %}
-          <div class="description inline truncate">
+          {% if bookmark_list.show_url %}
+            <div class="url-path truncate">
+              <a href="{{ bookmark_item.url }}" target="{{ bookmark_list.link_target }}" rel="noopener"
+                class="url-display">
+                {{ bookmark_item.url }}
+              </a>
+            </div>
+          {% endif %}
+          {% if bookmark_list.description_display == 'inline' %}
+            <div class="description inline truncate">
+              {% if bookmark_item.tag_names %}
+                <span class="tags">
+                  {% for tag_name in bookmark_item.tag_names %}
+                    <a href="?{% add_tag_to_query tag_name %}">{{ tag_name|hash_tag }}</a>
+                  {% endfor %}
+                </span>
+              {% endif %}
+              {% if bookmark_item.tag_names and bookmark_item.description %} | {% endif %}
+              {% if bookmark_item.description %}
+                <span>{{ bookmark_item.description }}</span>
+              {% endif %}
+            </div>
+          {% else %}
+            {% if bookmark_item.description %}
+              <div class="description separate">{{ bookmark_item.description }}</div>
+            {% endif %}
             {% if bookmark_item.tag_names %}
-              <span class="tags">
+              <div class="tags">
                 {% for tag_name in bookmark_item.tag_names %}
                   <a href="?{% add_tag_to_query tag_name %}">{{ tag_name|hash_tag }}</a>
                 {% endfor %}
+              </div>
+            {% endif %}
+          {% endif %}
+          {% if bookmark_item.notes %}
+            <div class="notes bg-gray text-gray-dark">
+              <div class="markdown">{% markdown bookmark_item.notes %}</div>
+            </div>
+          {% endif %}
+          <div class="actions text-gray">
+            {% if bookmark_item.display_date %}
+              {% if bookmark_item.web_archive_snapshot_url %}
+                <a href="{{ bookmark_item.web_archive_snapshot_url }}"
+                  title="Show snapshot on the Internet Archive Wayback Machine"
+                  target="{{ bookmark_list.link_target }}"
+                  rel="noopener">
+                  {{ bookmark_item.display_date }} ∞
+                </a>
+              {% else %}
+                <span>{{ bookmark_item.display_date }}</span>
+              {% endif %}
+              <span>|</span>
+            {% endif %}
+            {# View link is visible for both owned and shared bookmarks #}
+            {% if bookmark_list.show_view_action %}
+              <a ld-fetch="{% url 'bookmarks:details_modal' bookmark_item.id %}?return_url={{ bookmark_list.return_url|urlencode }}"
+                ld-on="click" ld-target="body|append"
+                href="{% url 'bookmarks:details' bookmark_item.id %}">View</a>
+            {% endif %}
+            {% if bookmark_item.is_editable %}
+              {# Bookmark owner actions #}
+              {% if bookmark_list.show_edit_action %}
+                <a href="{% url 'bookmarks:edit' bookmark_item.id %}?return_url={{ bookmark_list.return_url|urlencode }}">Edit</a>
+              {% endif %}
+              {% if bookmark_list.show_archive_action %}
+                {% if bookmark_item.is_archived %}
+                  <button type="submit" name="unarchive" value="{{ bookmark_item.id }}"
+                          class="btn btn-link btn-sm">Unarchive
+                  </button>
+                {% else %}
+                  <button type="submit" name="archive" value="{{ bookmark_item.id }}"
+                          class="btn btn-link btn-sm">Archive
+                  </button>
+                {% endif %}
+              {% endif %}
+              {% if bookmark_list.show_remove_action %}
+                <button ld-confirm-button type="submit" name="remove" value="{{ bookmark_item.id }}"
+                        class="btn btn-link btn-sm">Remove
+                </button>
+              {% endif %}
+            {% else %}
+              {# Shared bookmark actions #}
+              <span>Shared by
+                <a href="?{% replace_query_param user=bookmark_item.owner.username %}">{{ bookmark_item.owner.username }}</a>
               </span>
             {% endif %}
-            {% if bookmark_item.tag_names and bookmark_item.description %} | {% endif %}
-            {% if bookmark_item.description %}
-              <span>{{ bookmark_item.description }}</span>
+            {% if bookmark_item.has_extra_actions %}
+              <div class="extra-actions">
+                <span class="hide-sm">|</span>
+                {% if bookmark_item.show_mark_as_read %}
+                  <button type="submit" name="mark_as_read" value="{{ bookmark_item.id }}"
+                          class="btn btn-link btn-sm btn-icon"
+                          ld-confirm-button ld-confirm-icon="ld-icon-read" ld-confirm-question="Mark as read?">
+                    <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
+                      <use xlink:href="#ld-icon-unread"></use>
+                    </svg>
+                    Unread
+                  </button>
+                {% endif %}
+                {% if bookmark_item.show_unshare %}
+                  <button type="submit" name="unshare" value="{{ bookmark_item.id }}"
+                          class="btn btn-link btn-sm btn-icon"
+                          ld-confirm-button ld-confirm-icon="ld-icon-unshare" ld-confirm-question="Unshare?">
+                    <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
+                      <use xlink:href="#ld-icon-share"></use>
+                    </svg>
+                    Shared
+                  </button>
+                {% endif %}
+                {% if bookmark_item.show_notes_button %}
+                  <button type="button" class="btn btn-link btn-sm btn-icon toggle-notes">
+                    <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
+                      <use xlink:href="#ld-icon-note"></use>
+                    </svg>
+                    Notes
+                  </button>
+                {% endif %}
+              </div>
             {% endif %}
           </div>
-        {% else %}
-          {% if bookmark_item.description %}
-            <div class="description separate">
-              {{ bookmark_item.description }}
-            </div>
-          {% endif %}
-          {% if bookmark_item.tag_names %}
-            <div class="tags">
-              {% for tag_name in bookmark_item.tag_names %}
-                <a href="?{% add_tag_to_query tag_name %}">{{ tag_name|hash_tag }}</a>
-              {% endfor %}
-            </div>
-          {% endif %}
-        {% endif %}
-        {% if bookmark_item.notes %}
-          <div class="notes bg-gray text-gray-dark">
-            <div class="markdown">
-              {% markdown bookmark_item.notes %}
-            </div>
-          </div>
-        {% endif %}
-        <div class="actions text-gray">
-          {% if bookmark_item.display_date %}
-            {% if bookmark_item.web_archive_snapshot_url %}
-              <a href="{{ bookmark_item.web_archive_snapshot_url }}"
-                 title="Show snapshot on the Internet Archive Wayback Machine"
-                 target="{{ bookmark_list.link_target }}"
-                 rel="noopener">
-                {{ bookmark_item.display_date }} ∞
-              </a>
-            {% else %}
-              <span>{{ bookmark_item.display_date }}</span>
-            {% endif %}
-            <span>|</span>
-          {% endif %}
-          {# View link is visible for both owned and shared bookmarks #}
-          {% if bookmark_list.show_view_action %}
-            <a ld-modal
-               modal-url="{% url 'bookmarks:details_modal' bookmark_item.id %}?return_url={{ bookmark_list.return_url|urlencode }}"
-               href="{% url 'bookmarks:details' bookmark_item.id %}">View</a>
-          {% endif %}
-          {% if bookmark_item.is_editable %}
-            {# Bookmark owner actions #}
-            {% if bookmark_list.show_edit_action %}
-              <a href="{% url 'bookmarks:edit' bookmark_item.id %}?return_url={{ bookmark_list.return_url|urlencode }}">Edit</a>
-            {% endif %}
-            {% if bookmark_list.show_archive_action %}
-              {% if bookmark_item.is_archived %}
-                <button type="submit" name="unarchive" value="{{ bookmark_item.id }}"
-                        class="btn btn-link btn-sm">Unarchive
-                </button>
-              {% else %}
-                <button type="submit" name="archive" value="{{ bookmark_item.id }}"
-                        class="btn btn-link btn-sm">Archive
-                </button>
-              {% endif %}
-            {% endif %}
-            {% if bookmark_list.show_remove_action %}
-              <button ld-confirm-button type="submit" name="remove" value="{{ bookmark_item.id }}"
-                      class="btn btn-link btn-sm">Remove
-              </button>
-            {% endif %}
-          {% else %}
-            {# Shared bookmark actions #}
-            <span>Shared by
-              <a href="?{% replace_query_param user=bookmark_item.owner.username %}">{{ bookmark_item.owner.username }}</a>
-            </span>
-          {% endif %}
-          {% if bookmark_item.has_extra_actions %}
-            <div class="extra-actions">
-              <span class="hide-sm">|</span>
-              {% if bookmark_item.show_mark_as_read %}
-                <button type="submit" name="mark_as_read" value="{{ bookmark_item.id }}"
-                        class="btn btn-link btn-sm btn-icon"
-                        ld-confirm-button confirm-icon="ld-icon-read" confirm-question="Mark as read?">
-                  <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
-                    <use xlink:href="#ld-icon-unread"></use>
-                  </svg>
-                  Unread
-                </button>
-              {% endif %}
-              {% if bookmark_item.show_unshare %}
-                <button type="submit" name="unshare" value="{{ bookmark_item.id }}"
-                        class="btn btn-link btn-sm btn-icon"
-                        ld-confirm-button confirm-icon="ld-icon-unshare" confirm-question="Unshare?">
-                  <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
-                    <use xlink:href="#ld-icon-share"></use>
-                  </svg>
-                  Shared
-                </button>
-              {% endif %}
-              {% if bookmark_item.show_notes_button %}
-                <button type="button" class="btn btn-link btn-sm btn-icon toggle-notes">
-                  <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
-                    <use xlink:href="#ld-icon-note"></use>
-                  </svg>
-                  Notes
-                </button>
-              {% endif %}
-            </div>
-          {% endif %}
         </div>
+        {% if bookmark_list.show_preview_images and bookmark_item.preview_image_file %}
+          <img class="preview-image" src="{% static bookmark_item.preview_image_file %}" loading="lazy"/>
+        {% endif %}
       </li>
     {% endfor %}
   </ul>

bookmarks/templates/bookmarks/bulk_edit/bar.html

@@ -2,7 +2,7 @@
 {% htmlmin %}
   <div class="bulk-edit-bar">
     <div class="bulk-edit-actions bg-gray">
-      <label ld-bulk-edit-checkbox all class="form-checkbox">
+      <label class="form-checkbox bulk-edit-checkbox all">
         <input type="checkbox">
         <i class="form-icon"></i>
       </label>

bookmarks/templates/bookmarks/bulk_edit/toggle.html

@@ -1,4 +1,4 @@
-<button ld-bulk-edit-active-toggle class="btn hide-sm ml-2" title="Bulk edit">
+<button class="btn hide-sm ml-2 bulk-edit-active-toggle" title="Bulk edit">
   <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" width="20px"
        height="20px">
     <path

bookmarks/templates/bookmarks/details/assets.html

@@ -1,37 +1,48 @@
-{% if details.assets %}
-  <div class="assets">
-    {% for asset in details.assets %}
-      <div class="asset" data-asset-id="{{ asset.id }}">
-        <div class="asset-icon {{ asset.icon_classes }}">
-          {% include 'bookmarks/details/asset_icon.html' %}
+<div {% if details.has_pending_assets %}
+  ld-fetch="{% url 'bookmarks:details_assets' details.bookmark.id %}"
+  ld-interval="5" ld-target="self|outerHTML"
+{% endif %}>
+  {% if details.assets %}
+    <div class="assets">
+      {% for asset in details.assets %}
+        <div class="asset" data-asset-id="{{ asset.id }}">
+          <div class="asset-icon {{ asset.icon_classes }}">
+            {% include 'bookmarks/details/asset_icon.html' %}
+          </div>
+          <div class="asset-text {{ asset.text_classes }}">
+          <span class="truncate">
+            {{ asset.display_name }}
+            {% if asset.status == 'pending' %}(queued){% endif %}
+            {% if asset.status == 'failure' %}(failed){% endif %}
+          </span>
+            {% if asset.file_size %}
+              <span class="filesize">{{ asset.file_size|filesizeformat }}</span>
+            {% endif %}
+          </div>
+          <div class="asset-actions">
+            {% if asset.file %}
+              <a class="btn btn-link" href="{% url 'bookmarks:assets.view' asset.id %}" target="_blank">View</a>
+            {% endif %}
+            {% if details.is_editable %}
+              <button ld-confirm-button type="submit" name="remove_asset" value="{{ asset.id }}" class="btn btn-link">
+                Remove
+              </button>
+            {% endif %}
+          </div>
         </div>
-        <div class="asset-text truncate {{ asset.text_classes }}">
-        <span>
-          {{ asset.display_name }}
-          {% if asset.status == 'pending' %}(queued){% endif %}
-          {% if asset.status == 'failure' %}(failed){% endif %}
-        </span>
-          {% if asset.file_size %}
-            <span class="filesize">{{ asset.file_size|filesizeformat }}</span>
-          {% endif %}
-        </div>
-        <div class="asset-actions">
-          {% if asset.file %}
-            <a class="btn btn-link" href="{% url 'bookmarks:assets.view' asset.id %}" target="_blank">View</a>
-          {% endif %}
-          {% if details.is_editable %}
-            <button ld-confirm-button type="submit" name="remove_asset" value="{{ asset.id }}" class="btn btn-link">
-              Remove
-            </button>
-          {% endif %}
-        </div>
-      </div>
-    {% endfor %}
-  </div>
-{% endif %}
+      {% endfor %}
+    </div>
+  {% endif %}
 
-{% if details.is_editable %}
-  <div class="assets-actions">
-    <button type="submit" name="create_snapshot" class="btn btn-link">Create HTML snapshot</button>
-  </div>
-{% endif %}
+  {% if details.is_editable %}
+    <div class="assets-actions">
+      <button type="submit" name="create_snapshot" class="btn btn-link"
+              {% if details.has_pending_assets %}disabled{% endif %}>Create HTML snapshot
+      </button>
+      <button ld-upload-button id="upload-asset" name="upload_asset" value="{{ details.bookmark.id }}" type="button"
+              class="btn btn-link">Upload file
+      </button>
+      <input id="upload-asset-file" name="upload_asset_file" type="file" class="d-hide">
+    </div>
+  {% endif %}
+</div>

bookmarks/templates/bookmarks/details/form.html

@@ -1,9 +1,8 @@
 {% load static %}
 {% load shared %}
 
-<form ld-form action="{% url 'bookmarks:details' details.bookmark.id %}"
-      refresh-url="{% url 'bookmarks:partials.details_form' details.bookmark.id %}"
-      refresh-events="bookmark-page-refresh"
+<form ld-form ld-fire="refresh-bookmark-list,refresh-tag-cloud,refresh-details"
+      action="{% url 'bookmarks:details' details.bookmark.id %}"
       method="post">
   <div class="weblinks">
     <a class="weblink" href="{{ details.bookmark.url }}" rel="noopener"
@@ -13,8 +12,19 @@
       {% endif %}
       <span>{{ details.bookmark.url }}</span>
     </a>
-    {% if details.bookmark.web_archive_snapshot_url %}
-      <a class="weblink" href="{{ details.bookmark.web_archive_snapshot_url }}"
+    {% if details.latest_snapshot %}
+      <a class="weblink" href="{% url 'bookmarks:assets.read' details.latest_snapshot.id %}"
+         target="{{ details.profile.bookmark_link_target }}">
+        {% if details.show_link_icons %}
+          <svg class="favicon" xmlns="http://www.w3.org/2000/svg">
+            <use xlink:href="#ld-icon-unread"></use>
+          </svg>
+        {% endif %}
+        <span>Reader mode</span>
+      </a>
+    {% endif %}
+    {% if details.web_archive_snapshot_url %}
+      <a class="weblink" href="{{ details.web_archive_snapshot_url }}"
          target="{{ details.profile.bookmark_link_target }}">
         {% if details.show_link_icons %}
           <svg class="favicon" viewBox="0 0 76 86" xmlns="http://www.w3.org/2000/svg">
@@ -23,10 +33,15 @@
                 fill="currentColor" fill-rule="evenodd"/>
           </svg>
         {% endif %}
-        <span>View on Internet Archive</span>
+        <span>Internet Archive</span>
       </a>
     {% endif %}
   </div>
+  {% if details.preview_image_enabled and details.bookmark.preview_image_file %}
+    <div class="preview-image">
+      <img src="{% static details.bookmark.preview_image_file %}"/>
+    </div>
+  {% endif %}
   <dl class="grid columns-2 columns-sm-1 gap-0">
     {% if details.is_editable %}
       <div class="status col-2">
@@ -35,14 +50,14 @@
           {% csrf_token %}
           <div class="form-group">
             <label class="form-switch">
-              <input ld-form-auto-submit type="checkbox" name="is_archived"
+              <input ld-auto-submit type="checkbox" name="is_archived"
                      {% if details.bookmark.is_archived %}checked{% endif %}>
               <i class="form-icon"></i> Archived
             </label>
           </div>
           <div class="form-group">
             <label class="form-switch">
-              <input ld-form-auto-submit type="checkbox" name="unread"
+              <input ld-auto-submit type="checkbox" name="unread"
                      {% if details.bookmark.unread %}checked{% endif %}>
               <i class="form-icon"></i> Unread
             </label>
@@ -50,7 +65,7 @@
           {% if details.profile.enable_sharing %}
             <div class="form-group">
               <label class="form-switch">
-                <input ld-form-auto-submit type="checkbox" name="shared"
+                <input ld-auto-submit type="checkbox" name="shared"
                        {% if details.bookmark.shared %}checked{% endif %}>
                 <i class="form-icon"></i> Shared
               </label>

bookmarks/templates/bookmarks/details_modal.html

@@ -1,4 +1,7 @@
-<div class="modal active bookmark-details">
+<div ld-modal
+     ld-fetch="{% url 'bookmarks:details_modal' details.bookmark.id %}" ld-on="refresh-details"
+     ld-select=".content" ld-target=".modal.bookmark-details .content|outerHTML"
+     class="modal active bookmark-details">
   <div class="modal-overlay" aria-label="Close"></div>
   <div class="modal-container">
     <div class="modal-header">

bookmarks/templates/bookmarks/form.html

@@ -23,10 +23,10 @@
     <label for="{{ form.tag_string.id_for_label }}" class="form-label">Tags</label>
     {{ form.tag_string|add_class:"form-input"|attr:"ld-tag-autocomplete"|attr:"autocomplete:off"|attr:"autocapitalize:off" }}
     <div class="form-input-hint">
-      Enter any number of tags separated by space and <strong>without</strong> the hash (#). If a tag does not
-      exist it will be
-      automatically created.
+      Enter any number of tags separated by space and <strong>without</strong> the hash (#).
+      If a tag does not exist it will be automatically created.
     </div>
+    <div class="form-input-hint auto-tags"></div>
     {{ form.tag_string.errors }}
   </div>
   <div class="form-group has-icon-right">
@@ -197,6 +197,18 @@
             } else {
               bookmarkExistsHint.style['display'] = 'none';
             }
+
+            // Preview auto tags
+            const autoTags = data.auto_tags;
+            const autoTagsHint = document.querySelector('.form-input-hint.auto-tags');
+
+            if (autoTags.length > 0) {
+              autoTags.sort();
+              autoTagsHint.style['display'] = 'block';
+              autoTagsHint.innerHTML = `Auto tags: ${autoTags.join(" ")}`;
+            } else {
+              autoTagsHint.style['display'] = 'none';
+            }
           });
       }
 

bookmarks/templates/bookmarks/index.html

@@ -4,11 +4,7 @@
 {% load bookmarks %}
 
 {% block content %}
-  <div class="bookmarks-page grid columns-md-1"
-       ld-bulk-edit
-       ld-bookmark-page
-       bookmarks-url="{% url 'bookmarks:partials.bookmark_list.active' %}"
-       tags-url="{% url 'bookmarks:partials.tag_cloud.active' %}">
+  <div ld-bulk-edit class="bookmarks-page grid columns-md-1">
 
     {# Bookmark list #}
     <section class="content-area col-2">
@@ -17,17 +13,22 @@
         <div class="header-controls">
           {% bookmark_search bookmark_list.search tag_cloud.tags %}
           {% include 'bookmarks/bulk_edit/toggle.html' %}
-          <button ld-modal modal-content=".tag-cloud" class="btn ml-2 show-md">Tags</button>
+          <button ld-fetch="{{ bookmark_list.tag_modal_url }}" ld-target="body|append" ld-on="click"
+                  class="btn ml-2 show-md">Tags
+          </button>
         </div>
       </div>
 
-      <form class="bookmark-actions"
+      <form ld-form ld-fire="refresh-bookmark-list,refresh-tag-cloud"
+            class="bookmark-actions"
             action="{{ bookmark_list.action_url|safe }}"
             method="post" autocomplete="off">
         {% csrf_token %}
         {% include 'bookmarks/bulk_edit/bar.html' with disable_actions='bulk_unarchive' %}
 
-        <div class="bookmark-list-container">
+        <div ld-fetch="{{ bookmark_list.refresh_url }}" ld-on="refresh-bookmark-list"
+             ld-fire="refresh-bookmark-list-done"
+             class="bookmark-list-container">
           {% include 'bookmarks/bookmark_list.html' %}
         </div>
       </form>
@@ -38,7 +39,8 @@
       <div class="content-area-header">
         <h2>Tags</h2>
       </div>
-      <div class="tag-cloud-container">
+      <div ld-fetch="{{ tag_cloud.refresh_url }}" ld-on="refresh-tag-cloud"
+           class="tag-cloud-container">
         {% include 'bookmarks/tag_cloud.html' %}
       </div>
     </section>

bookmarks/templates/bookmarks/read.html

@@ -0,0 +1,83 @@
+{% load sass_tags %}
+{% load static %}
+<!DOCTYPE html>
+<html lang="en" class="reader-mode">
+<head>
+  <meta charset="UTF-8">
+  <title>Reader view</title>
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, minimal-ui">
+  {% if request.user_profile.theme == 'light' %}
+    <link href="{% sass_src 'theme-light.scss' %}?v={{ app_version }}" rel="stylesheet" type="text/css"/>
+  {% elif request.user_profile.theme == 'dark' %}
+    <link href="{% sass_src 'theme-dark.scss' %}?v={{ app_version }}" rel="stylesheet" type="text/css"/>
+  {% else %}
+    {# Use auto theme as fallback #}
+    <link href="{% sass_src 'theme-dark.scss' %}?v={{ app_version }}" rel="stylesheet" type="text/css"
+          media="(prefers-color-scheme: dark)"/>
+    <link href="{% sass_src 'theme-light.scss' %}?v={{ app_version }}" rel="stylesheet" type="text/css"
+          media="(prefers-color-scheme: light)"/>
+  {% endif %}
+</head>
+<body>
+<template id="content">{{ content|safe }}</template>
+<script src="{% static 'vendor/Readability.js' %}" type="application/javascript"></script>
+<script type="application/javascript">
+  function estimateReadingTime(charCount, wordsPerMinute) {
+    const avgWordLength = 5;
+    const totalWords = charCount / avgWordLength;
+    return Math.ceil(totalWords / wordsPerMinute);
+  }
+
+  function postProcess(articleContent) {
+    articleContent.querySelectorAll('table').forEach(table => {
+      table.classList.add('table');
+    });
+  }
+
+  function makeReadable() {
+    const content = document.getElementById('content');
+    const contentHtml = content.innerHTML;
+    const dom = new DOMParser().parseFromString(contentHtml, 'text/html');
+    const article = new Readability(dom).parse();
+
+    document.title = article.title;
+
+    const container = document.createElement('div');
+    container.classList.add('container');
+
+    const articleTitle = document.createElement('h1');
+    articleTitle.textContent = article.title;
+    container.append(articleTitle);
+
+    const byline = [article.byline, article.siteName].filter(Boolean);
+    if (byline.length > 0) {
+      const articleByline = document.createElement('p');
+      articleByline.textContent = byline.join(' | ');
+      articleByline.classList.add('byline');
+      container.append(articleByline);
+    }
+
+    if(article.length) {
+      const minTime = estimateReadingTime(article.length, 225);
+      const maxTime = estimateReadingTime(article.length, 175);
+
+      const articleReadingTime = document.createElement('p');
+      articleReadingTime.textContent = `${minTime}-${maxTime} minutes`;
+      articleReadingTime.classList.add('reading-time');
+      container.append(articleReadingTime);
+    }
+
+    const divider = document.createElement('hr');
+    container.append(divider);
+
+    const articleContent = document.createElement('div');
+    articleContent.innerHTML = article.content;
+    postProcess(articleContent);
+    container.append(articleContent);
+
+    content.replaceWith(container);
+  }
+  makeReadable();
+</script>
+</body>
+</html>

bookmarks/templates/bookmarks/shared.html

@@ -4,10 +4,7 @@
 {% load bookmarks %}
 
 {% block content %}
-  <div class="bookmarks-page grid columns-md-1"
-       ld-bookmark-page
-       bookmarks-url="{% url 'bookmarks:partials.bookmark_list.shared' %}"
-       tags-url="{% url 'bookmarks:partials.tag_cloud.shared' %}">
+  <div class="bookmarks-page grid columns-md-1">
 
     {# Bookmark list #}
     <section class="content-area col-2">
@@ -15,15 +12,20 @@
         <h2>Shared bookmarks</h2>
         <div class="header-controls">
           {% bookmark_search bookmark_list.search tag_cloud.tags mode='shared' %}
-          <button ld-modal modal-content=".tag-cloud" class="btn ml-2 show-md">Tags</button>
+          <button ld-fetch="{{ bookmark_list.tag_modal_url }}" ld-target="body|append" ld-on="click"
+                  class="btn ml-2 show-md">Tags
+          </button>
         </div>
       </div>
 
-      <form class="bookmark-actions" action="{{ bookmark_list.action_url|safe }}"
-            method="post">
+      <form ld-form ld-fire="refresh-bookmark-list,refresh-tag-cloud"
+            class="bookmark-actions"
+            action="{{ bookmark_list.action_url|safe }}"
+            method="post" autocomplete="off">
         {% csrf_token %}
-
-        <div class="bookmark-list-container">
+        <div ld-fetch="{{ bookmark_list.refresh_url }}" ld-on="refresh-bookmark-list"
+             ld-fire="refresh-bookmark-list-done"
+             class="bookmark-list-container">
           {% include 'bookmarks/bookmark_list.html' %}
         </div>
       </form>
@@ -41,7 +43,8 @@
       <div class="content-area-header">
         <h2>Tags</h2>
       </div>
-      <div class="tag-cloud-container">
+      <div ld-fetch="{{ tag_cloud.refresh_url }}" ld-on="refresh-tag-cloud"
+           class="tag-cloud-container">
         {% include 'bookmarks/tag_cloud.html' %}
       </div>
     </section>

bookmarks/templates/bookmarks/tag_modal.html

@@ -0,0 +1,21 @@
+<div ld-modal class="modal active">
+  <div class="modal-overlay" aria-label="Close"></div>
+  <div class="modal-container">
+    <div class="modal-header d-flex justify-between align-center">
+      <div class="modal-title h5">Tags</div>
+      <button class="close">
+        <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" stroke-width="2"
+             stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
+          <path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
+          <path d="M18 6l-12 12"></path>
+          <path d="M6 6l12 12"></path>
+        </svg>
+      </button>
+    </div>
+    <div class="modal-body">
+      <div class="content">
+        {% include 'bookmarks/tag_cloud.html' %}
+      </div>
+    </div>
+  </div>
+</div>

bookmarks/templates/settings/general.html

@@ -8,6 +8,12 @@
 
     {# Profile section #}
     <section class="content-area">
+      {% if success_message %}
+        <div class="toast toast-success mb-4">{{ success_message }}</div>
+      {% endif %}
+      {% if error_message %}
+        <div class="toast toast-error mb-4">{{ error_message }}</div>
+      {% endif %}
       <h2>Profile</h2>
       <p>
         <a href="{% url 'change_password' %}">Change password</a>
@@ -104,6 +110,29 @@
             result will also include bookmarks where a search term matches otherwise.
           </div>
         </div>
+        <div class="form-group">
+          <label for="{{ form.tag_grouping.id_for_label }}" class="form-label">Tag grouping</label>
+          {{ form.tag_grouping|add_class:"form-select width-25 width-sm-100" }}
+          <div class="form-input-hint">
+            In alphabetical mode, tags will be grouped by the first letter.
+            If disabled, tags will not be grouped.
+          </div>
+        </div>
+        <div class="form-group">
+          <details {% if form.auto_tagging_rules.value %}open{% endif %}>
+            <summary>Auto Tagging</summary>
+            <label for="{{ form.auto_tagging_rules.id_for_label }}" class="text-assistive">Auto Tagging</label>
+            <div class="mt-2">
+              {{ form.auto_tagging_rules|add_class:"form-input monospace"|attr:"rows:6" }}
+            </div>
+          </details>
+          <div class="form-input-hint">
+            Automatically adds tags to bookmarks based on predefined rules.
+            Each line is a single rule that maps a URL to one or more tags. For example:
+            <pre>youtube.com video
+reddit.com/r/Music music reddit</pre>
+          </div>
+        </div>
         <div class="form-group">
           <label for="{{ form.enable_favicons.id_for_label }}" class="form-checkbox">
             {{ form.enable_favicons }}
@@ -111,6 +140,7 @@
           </label>
           <div class="form-input-hint">
             Automatically loads favicons for bookmarked websites and displays them next to each bookmark.
+            Enabling this feature automatically downloads all missing favicons.
             By default, this feature uses a <b>Google service</b> to download favicons.
             If you don't want to use this service, check the <a
               href="https://github.com/sissbruecker/linkding/blob/master/docs/Options.md#ld_favicon_provider"
@@ -120,13 +150,16 @@
           {% if request.user_profile.enable_favicons and enable_refresh_favicons %}
             <button class="btn mt-2" name="refresh_favicons">Refresh Favicons</button>
           {% endif %}
-          {% if refresh_favicons_success_message %}
-            <div class="has-success">
-              <p class="form-input-hint">
-                {{ refresh_favicons_success_message }}
-              </p>
-            </div>
-          {% endif %}
+        </div>
+        <div class="form-group">
+          <label for="{{ form.enable_preview_images.id_for_label }}" class="form-checkbox">
+            {{ form.enable_preview_images }}
+            <i class="form-icon"></i> Enable Preview Images
+          </label>
+          <div class="form-input-hint">
+            Automatically loads preview images for bookmarked websites and displays them next to each bookmark.
+            Enabling this feature automatically downloads all missing preview images.
+          </div>
         </div>
         <div class="form-group">
           <label for="{{ form.web_archive_integration.id_for_label }}" class="form-label">Internet Archive
@@ -173,14 +206,26 @@
               Automatically creates HTML snapshots when adding bookmarks. Alternatively, when disabled, snapshots can be
               created manually in the details view of a bookmark.
             </div>
+            <button class="btn mt-2" name="create_missing_html_snapshots">Create missing HTML snapshots</button>
           </div>
         {% endif %}
+        <div class="form-group">
+          <label for="{{ form.default_mark_unread.id_for_label }}" class="form-checkbox">
+            {{ form.default_mark_unread }}
+            <i class="form-icon"></i> Create bookmarks as unread by default
+          </label>
+          <div class="form-input-hint">
+            Sets the default state for the "Mark as unread" option when creating a new bookmark.
+            Setting this option will make all new bookmarks default to unread.
+            This can be overridden when creating each new bookmark.
+          </div>
+        </div>
         <div class="form-group">
           <details {% if form.custom_css.value %}open{% endif %}>
             <summary>Custom CSS</summary>
             <label for="{{ form.custom_css.id_for_label }}" class="text-assistive">Custom CSS</label>
             <div class="mt-2">
-              {{ form.custom_css|add_class:"form-input custom-css"|attr:"rows:6" }}
+              {{ form.custom_css|add_class:"form-input monospace"|attr:"rows:6" }}
             </div>
           </details>
           <div class="form-input-hint">
@@ -189,13 +234,6 @@
         </div>
         <div class="form-group">
           <input type="submit" name="update_profile" value="Save" class="btn btn-primary mt-2">
-          {% if update_profile_success_message %}
-            <div class="has-success">
-              <p class="form-input-hint">
-                {{ update_profile_success_message }}
-              </p>
-            </div>
-          {% endif %}
         </div>
       </form>
     </section>
@@ -224,20 +262,6 @@
             <input class="form-input" type="file" name="import_file">
             <input type="submit" class="input-group-btn btn btn-primary" value="Upload">
           </div>
-          {% if import_success_message %}
-            <div class="has-success">
-              <p class="form-input-hint">
-                {{ import_success_message }}
-              </p>
-            </div>
-          {% endif %}
-          {% if import_errors_message %}
-            <div class="has-error">
-              <p class="form-input-hint">
-                {{ import_errors_message }}
-              </p>
-            </div>
-          {% endif %}
         </div>
       </form>
     </section>

bookmarks/tests/helpers.py

@@ -39,6 +39,7 @@ class BookmarkFactoryMixin:
         website_description: str = "",
         web_archive_snapshot_url: str = "",
         favicon_file: str = "",
+        preview_image_file: str = "",
         added: datetime = None,
     ):
         if title is None:
@@ -67,6 +68,7 @@ class BookmarkFactoryMixin:
             shared=shared,
             web_archive_snapshot_url=web_archive_snapshot_url,
             favicon_file=favicon_file,
+            preview_image_file=preview_image_file,
         )
         bookmark.save()
         for tag in tags:
@@ -85,6 +87,8 @@ class BookmarkFactoryMixin:
         shared: bool = False,
         with_tags: bool = False,
         with_web_archive_snapshot_url: bool = False,
+        with_favicon_file: bool = False,
+        with_preview_image_file: bool = False,
         user: User = None,
     ):
         user = user or self.get_or_create_test_user()
@@ -116,6 +120,12 @@ class BookmarkFactoryMixin:
             web_archive_snapshot_url = ""
             if with_web_archive_snapshot_url:
                 web_archive_snapshot_url = f"https://web.archive.org/web/{i}"
+            favicon_file = ""
+            if with_favicon_file:
+                favicon_file = f"favicon_{i}.png"
+            preview_image_file = ""
+            if with_preview_image_file:
+                preview_image_file = f"preview_image_{i}.png"
             bookmark = self.setup_bookmark(
                 url=url,
                 title=title,
@@ -124,6 +134,8 @@ class BookmarkFactoryMixin:
                 shared=shared,
                 tags=tags,
                 web_archive_snapshot_url=web_archive_snapshot_url,
+                favicon_file=favicon_file,
+                preview_image_file=preview_image_file,
                 user=user,
             )
             bookmarks.append(bookmark)

bookmarks/tests/test_auto_tagging.py

@@ -0,0 +1,179 @@
+from bookmarks.services import auto_tagging
+from django.test import TestCase
+
+
+class AutoTaggingTestCase(TestCase):
+    def test_auto_tag_by_domain(self):
+        script = """
+            example.com example
+            test.com test
+        """
+        url = "https://example.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["example"]))
+
+    def test_auto_tag_by_domain_ignores_case(self):
+        script = """
+            EXAMPLE.com example
+        """
+        url = "https://example.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["example"]))
+
+    def test_auto_tag_by_domain_should_add_all_tags(self):
+        script = """
+            example.com one two three
+        """
+        url = "https://example.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["one", "two", "three"]))
+
+    def test_auto_tag_by_domain_work_with_idn_domains(self):
+        script = """
+            रजिस्ट्री.भारत tag1
+        """
+        url = "https://www.xn--81bg3cc2b2bk5hb.xn--h2brj9c/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["tag1"]))
+
+        script = """
+            xn--81bg3cc2b2bk5hb.xn--h2brj9c tag1
+        """
+        url = "https://www.रजिस्ट्री.भारत/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["tag1"]))
+
+    def test_auto_tag_by_domain_and_path(self):
+        script = """
+            example.com/one one
+            example.com/two two
+            test.com test
+        """
+        url = "https://example.com/one/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["one"]))
+
+    def test_auto_tag_by_domain_and_path_ignores_case(self):
+        script = """
+            example.com/One one
+        """
+        url = "https://example.com/one/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["one"]))
+
+    def test_auto_tag_by_domain_and_path_matches_path_ltr(self):
+        script = """
+            example.com/one one
+            example.com/two two
+            test.com test
+        """
+        url = "https://example.com/one/two"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["one"]))
+
+    def test_auto_tag_by_domain_ignores_domain_in_path(self):
+        script = """
+            example.com example
+        """
+        url = "https://test.com/example.com"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set([]))
+
+    def test_auto_tag_by_domain_includes_subdomains(self):
+        script = """
+            example.com example
+            test.example.com test
+            some.example.com some
+        """
+        url = "https://test.example.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["example", "test"]))
+
+    def test_auto_tag_by_domain_matches_domain_rtl(self):
+        script = """
+            example.com example
+        """
+        url = "https://example.com.bad-website.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set([]))
+
+    def test_auto_tag_by_domain_ignores_schema(self):
+        script = """
+            https://example.com/ https
+            http://example.com/ http
+        """
+        url = "http://example.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["https", "http"]))
+
+    def test_auto_tag_by_domain_ignores_lines_with_no_tags(self):
+        script = """
+            example.com
+        """
+        url = "https://example.com/"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set([]))
+
+    def test_auto_tag_by_domain_path_and_qs(self):
+        script = """
+            example.com/page?a=b tag1     # true, matches a=b
+            example.com/page?a=c&c=d tag2 # true, matches both a=c and c=d
+            example.com/page?c=d&l=p tag3 # false, l=p doesn't exists
+            example.com/page?a=bb tag4    # false bb != b
+            example.com/page?a=b&a=c tag5 # true, matches both a=b and a=c
+            example.com/page?a=B tag6     # true, matches a=b because case insensitive
+            example.com/page?A=b tag7     # true, matches a=b because case insensitive
+        """
+        url = "https://example.com/page/some?z=x&a=b&v=b&c=d&o=p&a=c"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["tag1", "tag2", "tag5", "tag6", "tag7"]))
+
+    def test_auto_tag_by_domain_path_and_qs_with_empty_value(self):
+        script = """
+            example.com/page?a= tag1
+            example.com/page?b= tag2
+        """
+        url = "https://example.com/page/some?a=value"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["tag1"]))
+
+    def test_auto_tag_by_domain_path_and_qs_works_with_encoded_url(self):
+        script = """
+            example.com/page?a=йцу tag1
+            example.com/page?a=%D0%B9%D1%86%D1%83 tag2
+        """
+        url = "https://example.com/page?a=%D0%B9%D1%86%D1%83"
+
+        tags = auto_tagging.get_tags(script, url)
+
+        self.assertEqual(tags, set(["tag1", "tag2"]))

bookmarks/tests/test_bookmark_archived_view.py

@@ -94,15 +94,10 @@ class BookmarkArchivedViewTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
         )
 
     def assertBulkActionForm(self, response, url: str):
-        html = collapse_whitespace(response.content.decode())
-        needle = collapse_whitespace(
-            f"""
-            <form class="bookmark-actions"
-                action="{url}"
-                method="post" autocomplete="off">
-        """
-        )
-        self.assertIn(needle, html)
+        soup = self.make_soup(response.content.decode())
+        form = soup.select_one("form.bookmark-actions")
+        self.assertIsNotNone(form)
+        self.assertEqual(form.attrs["action"], url)
 
     def test_should_list_archived_and_user_owned_bookmarks(self):
         other_user = User.objects.create_user(

bookmarks/tests/test_bookmark_asset_view.py

@@ -34,12 +34,12 @@ class BookmarkAssetViewTestCase(TestCase, BookmarkFactoryMixin):
         asset = self.setup_asset(bookmark=bookmark, file=filename)
         return asset
 
-    def test_view_access(self):
+    def view_access_test(self, view_name: str):
         # own bookmark
         bookmark = self.setup_bookmark()
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 200)
 
         # other user's bookmark
@@ -47,14 +47,14 @@ class BookmarkAssetViewTestCase(TestCase, BookmarkFactoryMixin):
         bookmark = self.setup_bookmark(user=other_user)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # shared, sharing disabled
         bookmark = self.setup_bookmark(user=other_user, shared=True)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # unshared, sharing enabled
@@ -64,31 +64,31 @@ class BookmarkAssetViewTestCase(TestCase, BookmarkFactoryMixin):
         bookmark = self.setup_bookmark(user=other_user, shared=False)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # shared, sharing enabled
         bookmark = self.setup_bookmark(user=other_user, shared=True)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 200)
 
-    def test_view_access_guest_user(self):
+    def view_access_guest_user_test(self, view_name: str):
         self.client.logout()
 
         # unshared, sharing disabled
         bookmark = self.setup_bookmark()
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # shared, sharing disabled
         bookmark = self.setup_bookmark(shared=True)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # unshared, sharing enabled
@@ -98,14 +98,14 @@ class BookmarkAssetViewTestCase(TestCase, BookmarkFactoryMixin):
         bookmark = self.setup_bookmark(shared=False)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # shared, sharing enabled
         bookmark = self.setup_bookmark(shared=True)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # unshared, public sharing enabled
@@ -114,12 +114,24 @@ class BookmarkAssetViewTestCase(TestCase, BookmarkFactoryMixin):
         bookmark = self.setup_bookmark(shared=False)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 404)
 
         # shared, public sharing enabled
         bookmark = self.setup_bookmark(shared=True)
         asset = self.setup_asset_with_file(bookmark)
 
-        response = self.client.get(reverse("bookmarks:assets.view", args=[asset.id]))
+        response = self.client.get(reverse(view_name, args=[asset.id]))
         self.assertEqual(response.status_code, 200)
+
+    def test_view_access(self):
+        self.view_access_test("bookmarks:assets.view")
+
+    def test_view_access_guest_user(self):
+        self.view_access_guest_user_test("bookmarks:assets.view")
+
+    def test_reader_view_access(self):
+        self.view_access_test("bookmarks:assets.read")
+
+    def test_reader_view_access_guest_user(self):
+        self.view_access_guest_user_test("bookmarks:assets.read")

bookmarks/tests/test_bookmark_details_modal.py

@@ -1,11 +1,14 @@
+import datetime
+import re
 from unittest.mock import patch
 
+from django.core.files.uploadedfile import SimpleUploadedFile
 from django.test import TestCase, override_settings
 from django.urls import reverse
-from django.utils import formats
+from django.utils import formats, timezone
 
 from bookmarks.models import BookmarkAsset, UserProfile
-from bookmarks.services import tasks
+from bookmarks.services import bookmarks, tasks
 from bookmarks.tests.helpers import BookmarkFactoryMixin, HtmlTestMixin
 
 
@@ -45,6 +48,9 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
     def find_weblink(self, soup, url):
         return soup.find("a", {"class": "weblink", "href": url})
 
+    def count_weblinks(self, soup):
+        return len(soup.find_all("a", {"class": "weblink"}))
+
     def find_asset(self, soup, asset):
         return soup.find("div", {"data-asset-id": asset.id})
 
@@ -105,17 +111,11 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
     def test_access_with_sharing(self):
         self.details_route_sharing_access_test(self.get_view_name(), True)
 
-    def test_form_partial_access(self):
-        # form partial is only used when submitting forms, which should be only
-        # accessible to the owner of the bookmark. As such assume it requires
-        # login.
-        self.details_route_access_test("bookmarks:partials.details_form", False)
+    def test_assets_access(self):
+        self.details_route_access_test("bookmarks:details_assets", True)
 
-    def test_form_partial_access_with_sharing(self):
-        # form partial is only used when submitting forms, which should be only
-        # accessible to the owner of the bookmark. As such assume it requires
-        # login.
-        self.details_route_sharing_access_test("bookmarks:partials.details_form", False)
+    def test_assets_access_with_sharing(self):
+        self.details_route_sharing_access_test("bookmarks:details_assets", True)
 
     def test_displays_title(self):
         # with title
@@ -177,20 +177,55 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
         self.assertIsNotNone(image)
         self.assertEqual(image["src"], "/static/example.png")
 
-    def test_internet_archive_link(self):
-        # without snapshot url
+    def test_reader_mode_link(self):
+        # no latest snapshot
         bookmark = self.setup_bookmark()
         soup = self.get_details(bookmark)
-        link = self.find_weblink(soup, bookmark.web_archive_snapshot_url)
-        self.assertIsNone(link)
+        self.assertEqual(self.count_weblinks(soup), 2)
 
-        # with snapshot url
+        # snapshot is not complete
+        self.setup_asset(
+            bookmark,
+            asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+            status=BookmarkAsset.STATUS_PENDING,
+        )
+        self.setup_asset(
+            bookmark,
+            asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+            status=BookmarkAsset.STATUS_FAILURE,
+        )
+        soup = self.get_details(bookmark)
+        self.assertEqual(self.count_weblinks(soup), 2)
+
+        # not a snapshot
+        self.setup_asset(
+            bookmark,
+            asset_type="upload",
+            status=BookmarkAsset.STATUS_COMPLETE,
+        )
+        soup = self.get_details(bookmark)
+        self.assertEqual(self.count_weblinks(soup), 2)
+
+        # snapshot is complete
+        asset = self.setup_asset(
+            bookmark,
+            asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+            status=BookmarkAsset.STATUS_COMPLETE,
+        )
+        soup = self.get_details(bookmark)
+        self.assertEqual(self.count_weblinks(soup), 3)
+
+        reader_mode_url = reverse("bookmarks:assets.read", args=[asset.id])
+        link = self.find_weblink(soup, reader_mode_url)
+        self.assertIsNotNone(link)
+
+    def test_internet_archive_link_with_snapshot_url(self):
         bookmark = self.setup_bookmark(web_archive_snapshot_url="https://example.com/")
         soup = self.get_details(bookmark)
         link = self.find_weblink(soup, bookmark.web_archive_snapshot_url)
         self.assertIsNotNone(link)
         self.assertEqual(link["href"], bookmark.web_archive_snapshot_url)
-        self.assertEqual(link.text.strip(), "View on Internet Archive")
+        self.assertEqual(link.text.strip(), "Internet Archive")
 
         # favicons disabled
         bookmark = self.setup_bookmark(
@@ -223,6 +258,21 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
         image = link.select_one("svg")
         self.assertIsNotNone(image)
 
+    def test_internet_archive_link_with_fallback_url(self):
+        date_added = timezone.datetime(
+            2023, 8, 11, 21, 45, 11, tzinfo=datetime.timezone.utc
+        )
+        bookmark = self.setup_bookmark(url="https://example.com/", added=date_added)
+        fallback_web_archive_url = (
+            "https://web.archive.org/web/20230811214511/https://example.com/"
+        )
+
+        soup = self.get_details(bookmark)
+        link = self.find_weblink(soup, fallback_web_archive_url)
+        self.assertIsNotNone(link)
+        self.assertEqual(link["href"], fallback_web_archive_url)
+        self.assertEqual(link.text.strip(), "Internet Archive")
+
     def test_weblinks_respect_target_setting(self):
         bookmark = self.setup_bookmark(web_archive_snapshot_url="https://example.com/")
 
@@ -259,6 +309,36 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
             web_archive_link["target"], UserProfile.BOOKMARK_LINK_TARGET_SELF
         )
 
+    def test_preview_image(self):
+        # without image
+        bookmark = self.setup_bookmark()
+        soup = self.get_details(bookmark)
+        image = soup.select_one("div.preview-image img")
+        self.assertIsNone(image)
+
+        # with image
+        bookmark = self.setup_bookmark(preview_image_file="example.png")
+        soup = self.get_details(bookmark)
+        image = soup.select_one("div.preview-image img")
+        self.assertIsNone(image)
+
+        # preview images enabled, no image
+        profile = self.get_or_create_test_user().profile
+        profile.enable_preview_images = True
+        profile.save()
+
+        bookmark = self.setup_bookmark()
+        soup = self.get_details(bookmark)
+        image = soup.select_one("div.preview-image img")
+        self.assertIsNone(image)
+
+        # preview images enabled, image present
+        bookmark = self.setup_bookmark(preview_image_file="example.png")
+        soup = self.get_details(bookmark)
+        image = soup.select_one("div.preview-image img")
+        self.assertIsNotNone(image)
+        self.assertEqual(image["src"], "/static/example.png")
+
     def test_status(self):
         # renders form
         bookmark = self.setup_bookmark()
@@ -765,6 +845,27 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
         self.assertEqual(response.status_code, 404)
         self.assertTrue(BookmarkAsset.objects.filter(id=asset.id).exists())
 
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_assets_refresh_when_having_pending_asset(self):
+        bookmark = self.setup_bookmark()
+        asset = self.setup_asset(bookmark, status=BookmarkAsset.STATUS_COMPLETE)
+        fetch_url = reverse("bookmarks:details_assets", args=[bookmark.id])
+
+        # no pending asset
+        soup = self.get_details(bookmark)
+        files_section = self.find_section(soup, "Files")
+        assets_wrapper = files_section.find("div", {"ld-fetch": fetch_url})
+        self.assertIsNone(assets_wrapper)
+
+        # with pending asset
+        asset.status = BookmarkAsset.STATUS_PENDING
+        asset.save()
+
+        soup = self.get_details(bookmark)
+        files_section = self.find_section(soup, "Files")
+        assets_wrapper = files_section.find("div", {"ld-fetch": fetch_url})
+        self.assertIsNotNone(assets_wrapper)
+
     @override_settings(LD_ENABLE_SNAPSHOTS=True)
     def test_create_snapshot(self):
         with patch.object(
@@ -777,3 +878,58 @@ class BookmarkDetailsModalTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin
             self.assertEqual(response.status_code, 302)
 
             self.assertEqual(bookmark.bookmarkasset_set.count(), 1)
+
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_create_snapshot_is_disabled_when_having_pending_asset(self):
+        bookmark = self.setup_bookmark()
+        asset = self.setup_asset(bookmark, status=BookmarkAsset.STATUS_COMPLETE)
+
+        # no pending asset
+        soup = self.get_details(bookmark)
+        files_section = self.find_section(soup, "Files")
+        create_button = files_section.find(
+            "button", string=re.compile("Create HTML snapshot")
+        )
+        self.assertFalse(create_button.has_attr("disabled"))
+
+        # with pending asset
+        asset.status = BookmarkAsset.STATUS_PENDING
+        asset.save()
+
+        soup = self.get_details(bookmark)
+        files_section = self.find_section(soup, "Files")
+        create_button = files_section.find(
+            "button", string=re.compile("Create HTML snapshot")
+        )
+        self.assertTrue(create_button.has_attr("disabled"))
+
+    def test_upload_file(self):
+        bookmark = self.setup_bookmark()
+        file_content = b"file content"
+        upload_file = SimpleUploadedFile("test.txt", file_content)
+
+        with patch.object(bookmarks, "upload_asset") as mock_upload_asset:
+            response = self.client.post(
+                self.get_base_url(bookmark),
+                {"upload_asset": "", "upload_asset_file": upload_file},
+            )
+            self.assertEqual(response.status_code, 302)
+
+            mock_upload_asset.assert_called_once()
+
+            args, kwargs = mock_upload_asset.call_args
+            self.assertEqual(args[0], bookmark)
+
+            upload_file = args[1]
+            self.assertEqual(upload_file.name, "test.txt")
+
+    def test_upload_file_without_file(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(bookmarks, "upload_asset") as mock_upload_asset:
+            response = self.client.post(
+                self.get_base_url(bookmark),
+                {"upload_asset": ""},
+            )
+            self.assertEqual(response.status_code, 400)
+            mock_upload_asset.assert_not_called()

bookmarks/tests/test_bookmark_index_view.py

@@ -6,11 +6,7 @@ from django.test import TestCase
 from django.urls import reverse
 
 from bookmarks.models import Bookmark, BookmarkSearch, Tag, UserProfile
-from bookmarks.tests.helpers import (
-    BookmarkFactoryMixin,
-    HtmlTestMixin,
-    collapse_whitespace,
-)
+from bookmarks.tests.helpers import BookmarkFactoryMixin, HtmlTestMixin
 
 
 class BookmarkIndexViewTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
@@ -94,15 +90,10 @@ class BookmarkIndexViewTestCase(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
         )
 
     def assertBulkActionForm(self, response, url: str):
-        html = collapse_whitespace(response.content.decode())
-        needle = collapse_whitespace(
-            f"""
-            <form class="bookmark-actions"
-                action="{url}"
-                method="post" autocomplete="off">
-        """
-        )
-        self.assertIn(needle, html)
+        soup = self.make_soup(response.content.decode())
+        form = soup.select_one("form.bookmark-actions")
+        self.assertIsNotNone(form)
+        self.assertEqual(form.attrs["action"], url)
 
     def test_should_list_unarchived_and_user_owned_bookmarks(self):
         other_user = User.objects.create_user(

bookmarks/tests/test_bookmark_new_view.py

@@ -210,3 +210,25 @@ class BookmarkNewViewTestCase(TestCase, BookmarkFactoryMixin):
         response = self.client.get(reverse("bookmarks:edit", args=[bookmark.id]))
 
         self.assertContains(response, '<details class="notes">', count=1)
+
+    def test_should_not_check_unread_by_default(self):
+        response = self.client.get(reverse("bookmarks:new"))
+        html = response.content.decode()
+
+        self.assertInHTML(
+            '<input type="checkbox" name="unread" id="id_unread">',
+            html,
+        )
+
+    def test_should_check_unread_when_configured_in_profile(self):
+        self.user.profile.default_mark_unread = True
+        self.user.profile.save()
+
+        response = self.client.get(reverse("bookmarks:new"))
+        html = response.content.decode()
+
+        self.assertInHTML(
+            '<input type="checkbox" name="unread" value="true" '
+            'id="id_unread" checked="">',
+            html,
+        )

bookmarks/tests/test_bookmarks_api.py

@@ -36,6 +36,16 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
             expectation["website_title"] = bookmark.website_title
             expectation["website_description"] = bookmark.website_description
             expectation["web_archive_snapshot_url"] = bookmark.web_archive_snapshot_url
+            expectation["favicon_url"] = (
+                f"http://testserver/static/{bookmark.favicon_file}"
+                if bookmark.favicon_file
+                else None
+            )
+            expectation["preview_image_url"] = (
+                f"http://testserver/static/{bookmark.preview_image_file}"
+                if bookmark.preview_image_file
+                else None
+            )
             expectation["is_archived"] = bookmark.is_archived
             expectation["unread"] = bookmark.unread
             expectation["shared"] = bookmark.shared
@@ -65,7 +75,11 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
     def test_list_bookmarks_with_more_details(self):
         self.authenticate()
         bookmarks = self.setup_numbered_bookmarks(
-            5, with_tags=True, with_web_archive_snapshot_url=True
+            5,
+            with_tags=True,
+            with_web_archive_snapshot_url=True,
+            with_favicon_file=True,
+            with_preview_image_file=True,
         )
 
         response = self.get(
@@ -171,6 +185,23 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
         )
         self.assertBookmarkListEqual(response.data["results"], archived_bookmarks)
 
+    def test_list_archived_bookmarks_with_more_details(self):
+        self.authenticate()
+        archived_bookmarks = self.setup_numbered_bookmarks(
+            5,
+            archived=True,
+            with_tags=True,
+            with_web_archive_snapshot_url=True,
+            with_favicon_file=True,
+            with_preview_image_file=True,
+        )
+
+        response = self.get(
+            reverse("bookmarks:bookmark-archived"),
+            expected_status_code=status.HTTP_200_OK,
+        )
+        self.assertBookmarkListEqual(response.data["results"], archived_bookmarks)
+
     def test_list_archived_bookmarks_should_filter_by_query(self):
         self.authenticate()
         search_value = self.get_random_string()
@@ -220,6 +251,26 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
         )
         self.assertBookmarkListEqual(response.data["results"], shared_bookmarks)
 
+    def test_list_shared_bookmarks_with_more_details(self):
+        self.authenticate()
+
+        other_user = self.setup_user(enable_sharing=True)
+        shared_bookmarks = self.setup_numbered_bookmarks(
+            5,
+            shared=True,
+            user=other_user,
+            with_tags=True,
+            with_web_archive_snapshot_url=True,
+            with_favicon_file=True,
+            with_preview_image_file=True,
+        )
+
+        response = self.get(
+            reverse("bookmarks:bookmark-shared"),
+            expected_status_code=status.HTTP_200_OK,
+        )
+        self.assertBookmarkListEqual(response.data["results"], shared_bookmarks)
+
     def test_list_only_publicly_shared_bookmarks_when_not_logged_in(self):
         user1 = self.setup_user(enable_sharing=True, enable_public_sharing=True)
         user2 = self.setup_user(enable_sharing=True)
@@ -440,6 +491,20 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
         bookmark = Bookmark.objects.get(url=data["url"])
         self.assertFalse(bookmark.shared)
 
+    def test_create_bookmark_should_add_tags_from_auto_tagging(self):
+        tag1 = self.setup_tag()
+        tag2 = self.setup_tag()
+
+        self.authenticate()
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = f"example.com {tag2.name}"
+        profile.save()
+
+        data = {"url": "https://example.com/", "tag_names": [tag1.name]}
+        self.post(reverse("bookmarks:bookmark-list"), data, status.HTTP_201_CREATED)
+        bookmark = Bookmark.objects.get(url=data["url"])
+        self.assertCountEqual(bookmark.tags.all(), [tag1, tag2])
+
     def test_get_bookmark(self):
         self.authenticate()
         bookmark = self.setup_bookmark()
@@ -512,6 +577,22 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
         updated_bookmark = Bookmark.objects.get(id=bookmark.id)
         self.assertEqual(updated_bookmark.shared, True)
 
+    def test_update_bookmark_adds_tags_from_auto_tagging(self):
+        bookmark = self.setup_bookmark()
+        tag1 = self.setup_tag()
+        tag2 = self.setup_tag()
+
+        self.authenticate()
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = f"example.com {tag2.name}"
+        profile.save()
+
+        data = {"url": "https://example.com/", "tag_names": [tag1.name]}
+        url = reverse("bookmarks:bookmark-detail", args=[bookmark.id])
+        self.put(url, data, expected_status_code=status.HTTP_200_OK)
+        updated_bookmark = Bookmark.objects.get(id=bookmark.id)
+        self.assertCountEqual(updated_bookmark.tags.all(), [tag1, tag2])
+
     def test_patch_bookmark(self):
         self.authenticate()
         bookmark = self.setup_bookmark()
@@ -583,6 +664,22 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
         self.assertEqual(updated_bookmark.description, bookmark.description)
         self.assertListEqual(updated_bookmark.tag_names, bookmark.tag_names)
 
+    def test_patch_bookmark_adds_tags_from_auto_tagging(self):
+        bookmark = self.setup_bookmark()
+        tag1 = self.setup_tag()
+        tag2 = self.setup_tag()
+
+        self.authenticate()
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = f"example.com {tag2.name}"
+        profile.save()
+
+        data = {"tag_names": [tag1.name]}
+        url = reverse("bookmarks:bookmark-detail", args=[bookmark.id])
+        self.patch(url, data, expected_status_code=status.HTTP_200_OK)
+        updated_bookmark = Bookmark.objects.get(id=bookmark.id)
+        self.assertCountEqual(updated_bookmark.tags.all(), [tag1, tag2])
+
     def test_delete_bookmark(self):
         self.authenticate()
         bookmark = self.setup_bookmark()
@@ -628,7 +725,10 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
             website_loader, "load_website_metadata"
         ) as mock_load_website_metadata:
             expected_metadata = WebsiteMetadata(
-                "https://example.com", "Scraped metadata", "Scraped description"
+                "https://example.com",
+                "Scraped metadata",
+                "Scraped description",
+                "https://example.com/preview.png",
             )
             mock_load_website_metadata.return_value = expected_metadata
 
@@ -640,9 +740,10 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
             metadata = response.data["metadata"]
 
             self.assertIsNotNone(metadata)
-            self.assertIsNotNone(expected_metadata.url, metadata["url"])
-            self.assertIsNotNone(expected_metadata.title, metadata["title"])
-            self.assertIsNotNone(expected_metadata.description, metadata["description"])
+            self.assertEqual(expected_metadata.url, metadata["url"])
+            self.assertEqual(expected_metadata.title, metadata["title"])
+            self.assertEqual(expected_metadata.description, metadata["description"])
+            self.assertEqual(expected_metadata.preview_image, metadata["preview_image"])
 
     def test_check_returns_bookmark_if_url_is_bookmarked(self):
         self.authenticate()
@@ -651,6 +752,8 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
             url="https://example.com",
             title="Example title",
             description="Example description",
+            favicon_file="favicon.png",
+            preview_image_file="preview.png",
         )
 
         url = reverse("bookmarks:bookmark-check")
@@ -665,6 +768,12 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
         self.assertEqual(bookmark.url, bookmark_data["url"])
         self.assertEqual(bookmark.title, bookmark_data["title"])
         self.assertEqual(bookmark.description, bookmark_data["description"])
+        self.assertEqual(
+            "http://testserver/static/favicon.png", bookmark_data["favicon_url"]
+        )
+        self.assertEqual(
+            "http://testserver/static/preview.png", bookmark_data["preview_image_url"]
+        )
 
     def test_check_returns_existing_metadata_if_url_is_bookmarked(self):
         self.authenticate()
@@ -687,9 +796,38 @@ class BookmarksApiTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
 
             mock_load_website_metadata.assert_not_called()
             self.assertIsNotNone(metadata)
-            self.assertIsNotNone(bookmark.url, metadata["url"])
-            self.assertIsNotNone(bookmark.website_title, metadata["title"])
-            self.assertIsNotNone(bookmark.website_description, metadata["description"])
+            self.assertEqual(bookmark.url, metadata["url"])
+            self.assertEqual(bookmark.website_title, metadata["title"])
+            self.assertEqual(bookmark.website_description, metadata["description"])
+            self.assertIsNone(metadata["preview_image"])
+
+    def test_check_returns_no_auto_tags_if_none_configured(self):
+        self.authenticate()
+
+        url = reverse("bookmarks:bookmark-check")
+        check_url = urllib.parse.quote_plus("https://example.com")
+        response = self.get(
+            f"{url}?url={check_url}", expected_status_code=status.HTTP_200_OK
+        )
+        auto_tags = response.data["auto_tags"]
+
+        self.assertCountEqual(auto_tags, [])
+
+    def test_check_returns_matching_auto_tags(self):
+        self.authenticate()
+
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = "example.com tag1 tag2"
+        profile.save()
+
+        url = reverse("bookmarks:bookmark-check")
+        check_url = urllib.parse.quote_plus("https://example.com")
+        response = self.get(
+            f"{url}?url={check_url}", expected_status_code=status.HTTP_200_OK
+        )
+        auto_tags = response.data["auto_tags"]
+
+        self.assertCountEqual(auto_tags, ["tag1", "tag2"])
 
     def test_can_only_access_own_bookmarks(self):
         self.authenticate()

bookmarks/tests/test_bookmarks_list_template.py

@@ -1,3 +1,4 @@
+import datetime
 from typing import Type
 
 from dateutil.relativedelta import relativedelta
@@ -20,7 +21,7 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
         self, html: str, bookmark: Bookmark, link_target: str = "_blank"
     ):
         favicon_img = (
-            f'<img src="/static/{bookmark.favicon_file}" alt="">'
+            f'<img class="favicon" src="/static/{bookmark.favicon_file}" alt="">'
             if bookmark.favicon_file
             else ""
         )
@@ -36,15 +37,6 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             html,
         )
 
-    def assertDateLabel(self, html: str, label_content: str):
-        self.assertInHTML(
-            f"""
-        <span>{label_content}</span>
-        <span>|</span>
-        """,
-            html,
-        )
-
     def assertWebArchiveLink(
         self, html: str, label_content: str, url: str, link_target: str = "_blank"
     ):
@@ -80,7 +72,9 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
         details_modal_url = reverse("bookmarks:details_modal", args=[bookmark.id])
         self.assertInHTML(
             f"""
-                <a ld-modal modal-url="{details_modal_url}?return_url={return_url}" href="{details_url}">View</a>
+                <a ld-fetch="{details_modal_url}?return_url={return_url}" 
+                   ld-on="click" ld-target="body|append" 
+                   href="{details_url}">View</a>
             """,
             html,
             count=count,
@@ -146,19 +140,41 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
         )
 
     def assertFaviconVisible(self, html: str, bookmark: Bookmark):
-        self.assertFaviconCount(html, bookmark, 1)
+        self.assertFavicon(html, bookmark, True)
 
     def assertFaviconHidden(self, html: str, bookmark: Bookmark):
-        self.assertFaviconCount(html, bookmark, 0)
+        self.assertFavicon(html, bookmark, False)
 
-    def assertFaviconCount(self, html: str, bookmark: Bookmark, count=1):
-        self.assertInHTML(
-            f"""
-            <img src="/static/{bookmark.favicon_file}" alt="">
-            """,
-            html,
-            count=count,
-        )
+    def assertFavicon(self, html: str, bookmark: Bookmark, visible=True):
+        soup = self.make_soup(html)
+
+        favicon = soup.select_one(".favicon")
+
+        if not visible:
+            self.assertIsNone(favicon)
+            return
+
+        url = f"/static/{bookmark.favicon_file}"
+        self.assertIsNotNone(favicon)
+        self.assertEqual(favicon["src"], url)
+
+    def assertPreviewImageVisible(self, html: str, bookmark: Bookmark):
+        self.assertPreviewImage(html, bookmark, True)
+
+    def assertPreviewImageHidden(self, html: str, bookmark: Bookmark):
+        self.assertPreviewImage(html, bookmark, False)
+
+    def assertPreviewImage(self, html: str, bookmark: Bookmark, visible=True):
+        soup = self.make_soup(html)
+        preview_image = soup.select_one(".preview-image")
+
+        if not visible:
+            self.assertIsNone(preview_image)
+            return
+
+        url = f"/static/{bookmark.preview_image_file}"
+        self.assertIsNotNone(preview_image)
+        self.assertEqual(preview_image["src"], url)
 
     def assertBookmarkURLCount(
         self, html: str, bookmark: Bookmark, link_target: str = "_blank", count=0
@@ -216,7 +232,7 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             f"""
         <button type="submit" name="unshare" value="{bookmark.id}"
                 class="btn btn-link btn-sm btn-icon"
-                ld-confirm-button confirm-icon="ld-icon-unshare" confirm-question="Unshare?">
+                ld-confirm-button ld-confirm-icon="ld-icon-unshare" ld-confirm-question="Unshare?">
           <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
             <use xlink:href="#ld-icon-share"></use>
           </svg>
@@ -232,7 +248,7 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             f"""
         <button type="submit" name="mark_as_read" value="{bookmark.id}"
                 class="btn btn-link btn-sm btn-icon"
-                ld-confirm-button confirm-icon="ld-icon-read" confirm-question="Mark as read?">
+                ld-confirm-button ld-confirm-icon="ld-icon-read" ld-confirm-question="Mark as read?">
           <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16">
             <use xlink:href="#ld-icon-unread"></use>
           </svg>
@@ -301,7 +317,7 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
         if has_description and has_tags:
             self.assertTrue("|" in description.text)
 
-        # contains description text
+        # contains description text, without leading/trailing whitespace
         if has_description:
             description_text = description.find("span", text=bookmark.description)
             self.assertIsNotNone(description_text)
@@ -370,10 +386,10 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             description = soup.select_one(".description")
             self.assertIsNone(description)
         else:
-            # contains description text
+            # contains description text, without leading/trailing whitespace
             description = soup.select_one(".description.separate")
             self.assertIsNotNone(description)
-            self.assertEqual(description.text.strip(), bookmark.description)
+            self.assertEqual(description.text, bookmark.description)
 
         if not has_tags:
             # no tags element
@@ -441,15 +457,6 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
         style = bookmark_list["style"]
         self.assertIn("--ld-bookmark-description-max-lines:3;", style)
 
-    def test_should_respect_absolute_date_setting(self):
-        bookmark = self.setup_date_format_test(
-            UserProfile.BOOKMARK_DATE_DISPLAY_ABSOLUTE
-        )
-        html = self.render_template()
-        formatted_date = formats.date_format(bookmark.date_added, "SHORT_DATE_FORMAT")
-
-        self.assertDateLabel(html, formatted_date)
-
     def test_should_render_web_archive_link_with_absolute_date_setting(self):
         bookmark = self.setup_date_format_test(
             UserProfile.BOOKMARK_DATE_DISPLAY_ABSOLUTE,
@@ -462,12 +469,6 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             html, formatted_date, bookmark.web_archive_snapshot_url
         )
 
-    def test_should_respect_relative_date_setting(self):
-        self.setup_date_format_test(UserProfile.BOOKMARK_DATE_DISPLAY_RELATIVE)
-        html = self.render_template()
-
-        self.assertDateLabel(html, "1 week ago")
-
     def test_should_render_web_archive_link_with_relative_date_setting(self):
         bookmark = self.setup_date_format_test(
             UserProfile.BOOKMARK_DATE_DISPLAY_RELATIVE,
@@ -477,6 +478,27 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
 
         self.assertWebArchiveLink(html, "1 week ago", bookmark.web_archive_snapshot_url)
 
+    def test_should_render_generated_web_archive_link_without_saved_snapshot_url(self):
+        user = self.get_or_create_test_user()
+        user.profile.bookmark_date_display = UserProfile.BOOKMARK_DATE_DISPLAY_ABSOLUTE
+        user.profile.save()
+
+        date_added = timezone.datetime(
+            2023, 8, 11, 21, 45, 11, tzinfo=datetime.timezone.utc
+        )
+        bookmark = self.setup_bookmark(
+            url="https://example.com/article", added=date_added
+        )
+
+        html = self.render_template()
+        formatted_date = formats.date_format(bookmark.date_added, "SHORT_DATE_FORMAT")
+
+        self.assertWebArchiveLink(
+            html,
+            formatted_date,
+            "https://web.archive.org/web/20230811214511/https://example.com/article",
+        )
+
     def test_bookmark_link_target_should_be_blank_by_default(self):
         bookmark = self.setup_bookmark()
         html = self.render_template()
@@ -638,6 +660,36 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             html,
         )
 
+    def test_preview_image_should_be_visible_when_preview_images_enabled(self):
+        profile = self.get_or_create_test_user().profile
+        profile.enable_preview_images = True
+        profile.save()
+
+        bookmark = self.setup_bookmark(preview_image_file="preview.png")
+        html = self.render_template()
+
+        self.assertPreviewImageVisible(html, bookmark)
+
+    def test_preview_image_should_be_hidden_when_preview_images_disabled(self):
+        profile = self.get_or_create_test_user().profile
+        profile.enable_preview_images = False
+        profile.save()
+
+        bookmark = self.setup_bookmark(preview_image_file="preview.png")
+        html = self.render_template()
+
+        self.assertPreviewImageHidden(html, bookmark)
+
+    def test_preview_image_should_be_hidden_when_there_is_no_preview_image(self):
+        profile = self.get_or_create_test_user().profile
+        profile.enable_preview_images = True
+        profile.save()
+
+        bookmark = self.setup_bookmark()
+        html = self.render_template()
+
+        self.assertPreviewImageHidden(html, bookmark)
+
     def test_favicon_should_be_visible_when_favicons_enabled(self):
         profile = self.get_or_create_test_user().profile
         profile.enable_favicons = True
@@ -782,10 +834,16 @@ class BookmarkListTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
 
     def test_note_cleans_html(self):
         self.setup_bookmark(notes='<script>alert("test")</script>')
+        self.setup_bookmark(
+            notes='<b ld-fetch="https://example.com" ld-on="click">bold text</b>'
+        )
         html = self.render_template()
 
         note_html = '&lt;script&gt;alert("test")&lt;/script&gt;'
-        self.assertNotes(html, note_html, 1)
+        self.assertIn(note_html, html, 1)
+
+        note_html = "<b>bold text</b>"
+        self.assertIn(note_html, html, 1)
 
     def test_notes_are_hidden_initially_by_default(self):
         self.setup_bookmark(notes="Test note")

bookmarks/tests/test_bookmarks_service.py

@@ -1,10 +1,13 @@
+import os
+import tempfile
 from unittest.mock import patch
 
 from django.contrib.auth import get_user_model
-from django.test import TestCase
+from django.core.files.uploadedfile import SimpleUploadedFile
+from django.test import TestCase, override_settings
 from django.utils import timezone
 
-from bookmarks.models import Bookmark, Tag
+from bookmarks.models import Bookmark, BookmarkAsset, Tag
 from bookmarks.services import tasks
 from bookmarks.services import website_loader
 from bookmarks.services.bookmarks import (
@@ -21,6 +24,7 @@ from bookmarks.services.bookmarks import (
     mark_bookmarks_as_unread,
     share_bookmarks,
     unshare_bookmarks,
+    upload_asset,
 )
 from bookmarks.services.website_loader import WebsiteMetadata
 from bookmarks.tests.helpers import BookmarkFactoryMixin
@@ -38,7 +42,10 @@ class BookmarkServiceTestCase(TestCase, BookmarkFactoryMixin):
             website_loader, "load_website_metadata"
         ) as mock_load_website_metadata:
             expected_metadata = WebsiteMetadata(
-                "https://example.com", "Website title", "Website description"
+                "https://example.com",
+                "Website title",
+                "Website description",
+                "https://example.com/preview.png",
             )
             mock_load_website_metadata.return_value = expected_metadata
 
@@ -123,6 +130,18 @@ class BookmarkServiceTestCase(TestCase, BookmarkFactoryMixin):
 
             mock_create_html_snapshot.assert_not_called()
 
+    def test_create_should_add_tags_from_auto_tagging(self):
+        tag1 = self.setup_tag()
+        tag2 = self.setup_tag()
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = f"example.com {tag2.name}"
+        profile.save()
+
+        bookmark_data = Bookmark(url="https://example.com")
+        bookmark = create_bookmark(bookmark_data, tag1.name, self.user)
+
+        self.assertCountEqual(bookmark.tags.all(), [tag1, tag2])
+
     def test_update_should_create_web_archive_snapshot_if_url_did_change(self):
         with patch.object(
             tasks, "create_web_archive_snapshot"
@@ -153,6 +172,7 @@ class BookmarkServiceTestCase(TestCase, BookmarkFactoryMixin):
                 "https://example.com/updated",
                 "Updated website title",
                 "Updated website description",
+                "https://example.com/preview.png",
             )
             mock_load_website_metadata.return_value = expected_metadata
 
@@ -193,6 +213,18 @@ class BookmarkServiceTestCase(TestCase, BookmarkFactoryMixin):
 
             mock_create_html_snapshot.assert_not_called()
 
+    def test_update_should_add_tags_from_auto_tagging(self):
+        tag1 = self.setup_tag()
+        tag2 = self.setup_tag()
+        profile = self.get_or_create_test_user().profile
+        profile.auto_tagging_rules = f"example.com {tag2.name}"
+        profile.save()
+        bookmark = self.setup_bookmark(url="https://example.com")
+
+        update_bookmark(bookmark, tag1.name, self.user)
+
+        self.assertCountEqual(bookmark.tags.all(), [tag1, tag2])
+
     def test_archive_bookmark(self):
         bookmark = Bookmark(
             url="https://example.com",
@@ -835,3 +867,50 @@ class BookmarkServiceTestCase(TestCase, BookmarkFactoryMixin):
         self.assertFalse(Bookmark.objects.get(id=bookmark1.id).shared)
         self.assertFalse(Bookmark.objects.get(id=bookmark2.id).shared)
         self.assertFalse(Bookmark.objects.get(id=bookmark3.id).shared)
+
+    def test_upload_asset_should_save_file(self):
+        bookmark = self.setup_bookmark()
+        with tempfile.TemporaryDirectory() as temp_assets:
+            with override_settings(LD_ASSET_FOLDER=temp_assets):
+                file_content = b"file content"
+                upload_file = SimpleUploadedFile(
+                    "test_file.txt", file_content, content_type="text/plain"
+                )
+                upload_asset(bookmark, upload_file)
+
+                assets = bookmark.bookmarkasset_set.all()
+                self.assertEqual(1, len(assets))
+
+                asset = assets[0]
+                self.assertEqual("test_file.txt", asset.display_name)
+                self.assertEqual("text/plain", asset.content_type)
+                self.assertEqual(upload_file.size, asset.file_size)
+                self.assertEqual(BookmarkAsset.STATUS_COMPLETE, asset.status)
+                self.assertTrue(asset.file.startswith("upload_"))
+                self.assertTrue(asset.file.endswith(upload_file.name))
+
+                # check file exists
+                filepath = os.path.join(temp_assets, asset.file)
+                self.assertTrue(os.path.exists(filepath))
+                with open(filepath, "rb") as f:
+                    self.assertEqual(file_content, f.read())
+
+    def test_upload_asset_should_be_failed_if_saving_file_fails(self):
+        bookmark = self.setup_bookmark()
+        # Use an invalid path to force an error
+        with override_settings(LD_ASSET_FOLDER="/non/existing/folder"):
+            file_content = b"file content"
+            upload_file = SimpleUploadedFile(
+                "test_file.txt", file_content, content_type="text/plain"
+            )
+            upload_asset(bookmark, upload_file)
+
+            assets = bookmark.bookmarkasset_set.all()
+            self.assertEqual(1, len(assets))
+
+            asset = assets[0]
+            self.assertEqual("test_file.txt", asset.display_name)
+            self.assertEqual("text/plain", asset.content_type)
+            self.assertIsNone(asset.file_size)
+            self.assertEqual(BookmarkAsset.STATUS_FAILURE, asset.status)
+            self.assertEqual("", asset.file)

bookmarks/tests/test_bookmarks_tasks.py

@@ -1,6 +1,4 @@
-import datetime
 import os.path
-from dataclasses import dataclass
 from unittest import mock
 
 import waybackpy
@@ -10,8 +8,6 @@ from django.test import TestCase, override_settings
 from huey.contrib.djhuey import HUEY as huey
 from waybackpy.exceptions import WaybackError
 
-import bookmarks.services.favicon_loader
-import bookmarks.services.wayback
 from bookmarks.models import BookmarkAsset, UserProfile
 from bookmarks.services import tasks, singlefile
 from bookmarks.tests.helpers import BookmarkFactoryMixin
@@ -29,12 +25,6 @@ def create_wayback_machine_save_api_mock(
     return mock_api
 
 
-@dataclass
-class MockCdxSnapshot:
-    archive_url: str
-    datetime_timestamp: datetime.datetime
-
-
 class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
 
     def setUp(self):
@@ -50,17 +40,6 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
         )
         self.mock_save_api_patcher.start()
 
-        self.mock_cdx_api = mock.Mock()
-        self.mock_cdx_api.newest.return_value = MockCdxSnapshot(
-            "https://example.com/newest_snapshot", datetime.datetime.now()
-        )
-        self.mock_cdx_api_patcher = mock.patch.object(
-            bookmarks.services.wayback,
-            "CustomWaybackMachineCDXServerAPI",
-            return_value=self.mock_cdx_api,
-        )
-        self.mock_cdx_api_patcher.start()
-
         self.mock_load_favicon_patcher = mock.patch(
             "bookmarks.services.favicon_loader.load_favicon"
         )
@@ -74,18 +53,25 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
             self.mock_singlefile_create_snapshot_patcher.start()
         )
 
+        self.mock_load_preview_image_patcher = mock.patch(
+            "bookmarks.services.preview_image_loader.load_preview_image"
+        )
+        self.mock_load_preview_image = self.mock_load_preview_image_patcher.start()
+        self.mock_load_preview_image.return_value = "preview_image.png"
+
         user = self.get_or_create_test_user()
         user.profile.web_archive_integration = (
             UserProfile.WEB_ARCHIVE_INTEGRATION_ENABLED
         )
         user.profile.enable_favicons = True
+        user.profile.enable_preview_images = True
         user.profile.save()
 
     def tearDown(self):
         self.mock_save_api_patcher.stop()
-        self.mock_cdx_api_patcher.stop()
         self.mock_load_favicon_patcher.stop()
         self.mock_singlefile_create_snapshot_patcher.stop()
+        self.mock_load_preview_image_patcher.stop()
         huey.storage.flush_results()
         huey.immediate = False
 
@@ -134,45 +120,6 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertEqual(bookmark.web_archive_snapshot_url, "https://other.com")
 
-    def test_create_web_archive_snapshot_should_use_newest_snapshot_as_fallback(self):
-        bookmark = self.setup_bookmark()
-        self.mock_save_api.save.side_effect = WaybackError
-
-        tasks.create_web_archive_snapshot(
-            self.get_or_create_test_user(), bookmark, False
-        )
-
-        bookmark.refresh_from_db()
-        self.mock_cdx_api.newest.assert_called_once()
-        self.assertEqual(
-            "https://example.com/newest_snapshot",
-            bookmark.web_archive_snapshot_url,
-        )
-
-    def test_create_web_archive_snapshot_should_ignore_missing_newest_snapshot(self):
-        bookmark = self.setup_bookmark()
-        self.mock_save_api.save.side_effect = WaybackError
-        self.mock_cdx_api.newest.return_value = None
-
-        tasks.create_web_archive_snapshot(
-            self.get_or_create_test_user(), bookmark, False
-        )
-
-        bookmark.refresh_from_db()
-        self.assertEqual("", bookmark.web_archive_snapshot_url)
-
-    def test_create_web_archive_snapshot_should_ignore_newest_snapshot_errors(self):
-        bookmark = self.setup_bookmark()
-        self.mock_save_api.save.side_effect = WaybackError
-        self.mock_cdx_api.newest.side_effect = WaybackError
-
-        tasks.create_web_archive_snapshot(
-            self.get_or_create_test_user(), bookmark, False
-        )
-
-        bookmark.refresh_from_db()
-        self.assertEqual("", bookmark.web_archive_snapshot_url)
-
     def test_create_web_archive_snapshot_should_not_save_stale_bookmark_data(self):
         bookmark = self.setup_bookmark()
 
@@ -195,69 +142,6 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
             bookmark.web_archive_snapshot_url,
         )
 
-    def test_load_web_archive_snapshot_should_update_snapshot_url(self):
-        bookmark = self.setup_bookmark()
-
-        tasks._load_web_archive_snapshot_task(bookmark.id)
-
-        bookmark.refresh_from_db()
-
-        self.assertEqual(self.executed_count(), 1)
-        self.mock_cdx_api.newest.assert_called_once()
-        self.assertEqual(
-            "https://example.com/newest_snapshot", bookmark.web_archive_snapshot_url
-        )
-
-    def test_load_web_archive_snapshot_should_handle_missing_bookmark_id(self):
-        tasks._load_web_archive_snapshot_task(123)
-
-        self.assertEqual(self.executed_count(), 1)
-        self.mock_cdx_api.newest.assert_not_called()
-
-    def test_load_web_archive_snapshot_should_skip_if_snapshot_exists(self):
-        bookmark = self.setup_bookmark(web_archive_snapshot_url="https://example.com")
-
-        tasks._load_web_archive_snapshot_task(bookmark.id)
-
-        self.assertEqual(self.executed_count(), 1)
-        self.mock_cdx_api.newest.assert_not_called()
-
-    def test_load_web_archive_snapshot_should_handle_missing_snapshot(self):
-        bookmark = self.setup_bookmark()
-        self.mock_cdx_api.newest.return_value = None
-
-        tasks._load_web_archive_snapshot_task(bookmark.id)
-
-        self.assertEqual("", bookmark.web_archive_snapshot_url)
-
-    def test_load_web_archive_snapshot_should_handle_wayback_errors(self):
-        bookmark = self.setup_bookmark()
-        self.mock_cdx_api.newest.side_effect = WaybackError
-
-        tasks._load_web_archive_snapshot_task(bookmark.id)
-
-        self.assertEqual("", bookmark.web_archive_snapshot_url)
-
-    def test_load_web_archive_snapshot_should_not_save_stale_bookmark_data(self):
-        bookmark = self.setup_bookmark()
-
-        # update bookmark during API call to check that saving
-        # the snapshot does not overwrite updated bookmark data
-        def mock_newest_impl():
-            bookmark.title = "Updated title"
-            bookmark.save()
-            return mock.DEFAULT
-
-        self.mock_cdx_api.newest.side_effect = mock_newest_impl
-
-        tasks._load_web_archive_snapshot_task(bookmark.id)
-        bookmark.refresh_from_db()
-
-        self.assertEqual("Updated title", bookmark.title)
-        self.assertEqual(
-            "https://example.com/newest_snapshot", bookmark.web_archive_snapshot_url
-        )
-
     @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
     def test_create_web_archive_snapshot_should_not_run_when_background_tasks_are_disabled(
         self,
@@ -284,59 +168,6 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertEqual(self.executed_count(), 0)
 
-    def test_schedule_bookmarks_without_snapshots_should_load_snapshot_for_all_bookmarks_without_snapshot(
-        self,
-    ):
-        user = self.get_or_create_test_user()
-        self.setup_bookmark()
-        self.setup_bookmark()
-        self.setup_bookmark()
-        self.setup_bookmark(web_archive_snapshot_url="https://example.com")
-        self.setup_bookmark(web_archive_snapshot_url="https://example.com")
-        self.setup_bookmark(web_archive_snapshot_url="https://example.com")
-
-        tasks.schedule_bookmarks_without_snapshots(user)
-
-        self.assertEqual(self.executed_count(), 4)
-        self.assertEqual(self.mock_cdx_api.newest.call_count, 3)
-
-    def test_schedule_bookmarks_without_snapshots_should_only_update_user_owned_bookmarks(
-        self,
-    ):
-        user = self.get_or_create_test_user()
-        other_user = User.objects.create_user(
-            "otheruser", "otheruser@example.com", "password123"
-        )
-        self.setup_bookmark()
-        self.setup_bookmark()
-        self.setup_bookmark()
-        self.setup_bookmark(user=other_user)
-        self.setup_bookmark(user=other_user)
-        self.setup_bookmark(user=other_user)
-
-        tasks.schedule_bookmarks_without_snapshots(user)
-
-        self.assertEqual(self.mock_cdx_api.newest.call_count, 3)
-
-    @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
-    def test_schedule_bookmarks_without_snapshots_should_not_run_when_background_tasks_are_disabled(
-        self,
-    ):
-        tasks.schedule_bookmarks_without_snapshots(self.user)
-
-        self.assertEqual(self.executed_count(), 0)
-
-    def test_schedule_bookmarks_without_snapshots_should_not_run_when_web_archive_integration_is_disabled(
-        self,
-    ):
-        self.user.profile.web_archive_integration = (
-            UserProfile.WEB_ARCHIVE_INTEGRATION_DISABLED
-        )
-        self.user.profile.save()
-        tasks.schedule_bookmarks_without_snapshots(self.user)
-
-        self.assertEqual(self.executed_count(), 0)
-
     def test_load_favicon_should_create_favicon_file(self):
         bookmark = self.setup_bookmark()
 
@@ -507,6 +338,136 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertEqual(self.executed_count(), 0)
 
+    def test_load_preview_image_should_create_preview_image_file(self):
+        bookmark = self.setup_bookmark()
+
+        tasks.load_preview_image(self.get_or_create_test_user(), bookmark)
+        bookmark.refresh_from_db()
+
+        self.assertEqual(self.executed_count(), 1)
+        self.assertEqual(bookmark.preview_image_file, "preview_image.png")
+
+    def test_load_preview_image_should_update_preview_image_file(self):
+        bookmark = self.setup_bookmark(
+            preview_image_file="preview_image.png",
+        )
+
+        self.mock_load_preview_image.return_value = "preview_image_upd.png"
+
+        tasks.load_preview_image(self.get_or_create_test_user(), bookmark)
+
+        bookmark.refresh_from_db()
+        self.mock_load_preview_image.assert_called_once()
+        self.assertEqual(bookmark.preview_image_file, "preview_image_upd.png")
+
+    def test_load_preview_image_should_set_blank_when_none_is_returned(self):
+        bookmark = self.setup_bookmark(
+            preview_image_file="preview_image.png",
+        )
+
+        self.mock_load_preview_image.return_value = None
+
+        tasks.load_preview_image(self.get_or_create_test_user(), bookmark)
+
+        bookmark.refresh_from_db()
+        self.mock_load_preview_image.assert_called_once()
+        self.assertEqual(bookmark.preview_image_file, "")
+
+    def test_load_preview_image_should_handle_missing_bookmark(self):
+        tasks._load_preview_image_task(123)
+
+        self.mock_load_preview_image.assert_not_called()
+
+    def test_load_preview_image_should_not_save_stale_bookmark_data(self):
+        bookmark = self.setup_bookmark()
+
+        # update bookmark during API call to check that saving
+        # the image does not overwrite updated bookmark data
+        def mock_load_preview_image_impl(url):
+            bookmark.title = "Updated title"
+            bookmark.save()
+            return "test.png"
+
+        self.mock_load_preview_image.side_effect = mock_load_preview_image_impl
+
+        tasks.load_preview_image(self.get_or_create_test_user(), bookmark)
+        bookmark.refresh_from_db()
+
+        self.assertEqual(bookmark.title, "Updated title")
+        self.assertEqual(bookmark.preview_image_file, "test.png")
+
+    @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
+    def test_load_preview_image_should_not_run_when_background_tasks_are_disabled(self):
+        bookmark = self.setup_bookmark()
+        tasks.load_preview_image(self.get_or_create_test_user(), bookmark)
+
+        self.assertEqual(self.executed_count(), 0)
+
+    def test_load_preview_image_should_not_run_when_preview_image_feature_is_disabled(
+        self,
+    ):
+        self.user.profile.enable_preview_images = False
+        self.user.profile.save()
+
+        bookmark = self.setup_bookmark()
+        tasks.load_preview_image(self.get_or_create_test_user(), bookmark)
+
+        self.assertEqual(self.executed_count(), 0)
+
+    def test_schedule_bookmarks_without_previews_should_load_preview_for_all_bookmarks_without_preview(
+        self,
+    ):
+        user = self.get_or_create_test_user()
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark(preview_image_file="test.png")
+        self.setup_bookmark(preview_image_file="test.png")
+        self.setup_bookmark(preview_image_file="test.png")
+
+        tasks.schedule_bookmarks_without_previews(user)
+
+        self.assertEqual(self.executed_count(), 4)
+        self.assertEqual(self.mock_load_preview_image.call_count, 3)
+
+    def test_schedule_bookmarks_without_previews_should_only_update_user_owned_bookmarks(
+        self,
+    ):
+        user = self.get_or_create_test_user()
+        other_user = User.objects.create_user(
+            "otheruser", "otheruser@example.com", "password123"
+        )
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark(user=other_user)
+        self.setup_bookmark(user=other_user)
+        self.setup_bookmark(user=other_user)
+
+        tasks.schedule_bookmarks_without_previews(user)
+
+        self.assertEqual(self.mock_load_preview_image.call_count, 3)
+
+    @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
+    def test_schedule_bookmarks_without_previews_should_not_run_when_background_tasks_are_disabled(
+        self,
+    ):
+        self.setup_bookmark()
+        tasks.schedule_bookmarks_without_previews(self.get_or_create_test_user())
+
+        self.assertEqual(self.executed_count(), 0)
+
+    def test_schedule_bookmarks_without_previews_should_not_run_when_preview_feature_is_disabled(
+        self,
+    ):
+        self.user.profile.enable_preview_images = False
+        self.user.profile.save()
+
+        self.setup_bookmark()
+        tasks.schedule_bookmarks_without_previews(self.get_or_create_test_user())
+
+        self.assertEqual(self.executed_count(), 0)
+
     @override_settings(LD_ENABLE_SNAPSHOTS=True)
     def test_create_html_snapshot_should_create_pending_asset(self):
         bookmark = self.setup_bookmark()
@@ -611,3 +572,89 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
         tasks.create_html_snapshot(bookmark)
 
         self.assertEqual(BookmarkAsset.objects.count(), 0)
+
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_create_missing_html_snapshots(self):
+        bookmarks_with_snapshots = []
+        bookmarks_without_snapshots = []
+
+        # setup bookmarks with snapshots
+        bookmark = self.setup_bookmark()
+        self.setup_asset(
+            bookmark=bookmark,
+            asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+            status=BookmarkAsset.STATUS_COMPLETE,
+        )
+        bookmarks_with_snapshots.append(bookmark)
+
+        bookmark = self.setup_bookmark()
+        self.setup_asset(
+            bookmark=bookmark,
+            asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+            status=BookmarkAsset.STATUS_PENDING,
+        )
+        bookmarks_with_snapshots.append(bookmark)
+
+        # setup bookmarks without snapshots
+        bookmark = self.setup_bookmark()
+        bookmarks_without_snapshots.append(bookmark)
+
+        bookmark = self.setup_bookmark()
+        self.setup_asset(
+            bookmark=bookmark,
+            asset_type=BookmarkAsset.TYPE_SNAPSHOT,
+            status=BookmarkAsset.STATUS_FAILURE,
+        )
+        bookmarks_without_snapshots.append(bookmark)
+
+        bookmark = self.setup_bookmark()
+        self.setup_asset(
+            bookmark=bookmark,
+            asset_type="some_other_type",
+            status=BookmarkAsset.STATUS_PENDING,
+        )
+        bookmarks_without_snapshots.append(bookmark)
+
+        bookmark = self.setup_bookmark()
+        self.setup_asset(
+            bookmark=bookmark,
+            asset_type="some_other_type",
+            status=BookmarkAsset.STATUS_COMPLETE,
+        )
+        bookmarks_without_snapshots.append(bookmark)
+
+        initial_assets = list(BookmarkAsset.objects.all())
+        initial_assets_count = len(initial_assets)
+        initial_asset_ids = [asset.id for asset in initial_assets]
+        count = tasks.create_missing_html_snapshots(self.get_or_create_test_user())
+
+        self.assertEqual(count, 4)
+        self.assertEqual(BookmarkAsset.objects.count(), initial_assets_count + count)
+
+        for bookmark in bookmarks_without_snapshots:
+            new_assets = BookmarkAsset.objects.filter(bookmark=bookmark).exclude(
+                id__in=initial_asset_ids
+            )
+            self.assertEqual(new_assets.count(), 1)
+
+        for bookmark in bookmarks_with_snapshots:
+            new_assets = BookmarkAsset.objects.filter(bookmark=bookmark).exclude(
+                id__in=initial_asset_ids
+            )
+            self.assertEqual(new_assets.count(), 0)
+
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_create_missing_html_snapshots_respects_current_user(self):
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark()
+
+        other_user = self.setup_user()
+        self.setup_bookmark(user=other_user)
+        self.setup_bookmark(user=other_user)
+        self.setup_bookmark(user=other_user)
+
+        count = tasks.create_missing_html_snapshots(self.get_or_create_test_user())
+
+        self.assertEqual(count, 3)
+        self.assertEqual(BookmarkAsset.objects.count(), count)

bookmarks/tests/test_favicon_loader.py

@@ -1,8 +1,9 @@
 import io
 import os.path
 import time
+import tempfile
 from pathlib import Path
-from unittest import mock, skip
+from unittest import mock
 
 from django.conf import settings
 from django.test import TestCase, override_settings
@@ -29,17 +30,21 @@ class MockStreamingResponse:
 
 class FaviconLoaderTestCase(TestCase):
     def setUp(self) -> None:
-        self.ensure_favicon_folder()
-        self.clear_favicon_folder()
+        self.temp_favicon_folder = tempfile.TemporaryDirectory()
+        self.favicon_folder_override = self.settings(
+            LD_FAVICON_FOLDER=self.temp_favicon_folder.name
+        )
+        self.favicon_folder_override.enable()
+
+    def tearDown(self) -> None:
+        self.temp_favicon_folder.cleanup()
+        self.favicon_folder_override.disable()
 
     def create_mock_response(self, icon_data=mock_icon_data, content_type="image/png"):
         mock_response = mock.Mock()
         mock_response.raw = io.BytesIO(icon_data)
         return MockStreamingResponse(icon_data, content_type)
 
-    def ensure_favicon_folder(self):
-        Path(settings.LD_FAVICON_FOLDER).mkdir(parents=True, exist_ok=True)
-
     def clear_favicon_folder(self):
         folder = Path(settings.LD_FAVICON_FOLDER)
         for file in folder.iterdir():
@@ -177,7 +182,6 @@ class FaviconLoaderTestCase(TestCase):
             self.assertTrue(self.icon_exists("https_example_com.png"))
 
         self.clear_favicon_folder()
-        self.ensure_favicon_folder()
 
         with mock.patch("requests.get") as mock_get:
             mock_get.return_value = self.create_mock_response(

bookmarks/tests/test_importer.py

@@ -444,17 +444,6 @@ class ImporterTestCase(TestCase, BookmarkFactoryMixin, ImportTestMixin):
         self.assertEqual(Bookmark.objects.all()[0].description, "Example description")
         self.assertEqual(Bookmark.objects.all()[0].notes, "Updated notes")
 
-    def test_schedule_snapshot_creation(self):
-        user = self.get_or_create_test_user()
-        test_html = self.render_html(tags_html="")
-
-        with patch.object(
-            tasks, "schedule_bookmarks_without_snapshots"
-        ) as mock_schedule_bookmarks_without_snapshots:
-            import_netscape_html(test_html, user)
-
-            mock_schedule_bookmarks_without_snapshots.assert_called_once_with(user)
-
     def test_schedule_favicon_loading(self):
         user = self.get_or_create_test_user()
         test_html = self.render_html(tags_html="")
@@ -465,3 +454,14 @@ class ImporterTestCase(TestCase, BookmarkFactoryMixin, ImportTestMixin):
             import_netscape_html(test_html, user)
 
             mock_schedule_bookmarks_without_favicons.assert_called_once_with(user)
+
+    def test_schedule_preview_loading(self):
+        user = self.get_or_create_test_user()
+        test_html = self.render_html(tags_html="")
+
+        with patch.object(
+            tasks, "schedule_bookmarks_without_previews"
+        ) as mock_schedule_bookmarks_without_previews:
+            import_netscape_html(test_html, user)
+
+            mock_schedule_bookmarks_without_previews.assert_called_once_with(user)

bookmarks/tests/test_preview_image_loader.py

@@ -0,0 +1,205 @@
+import io
+import os
+import tempfile
+from pathlib import Path
+from unittest import mock
+
+from django.conf import settings
+from django.test import TestCase
+
+from bookmarks.services import preview_image_loader
+
+mock_image_data = b"mock_image"
+
+
+class MockStreamingResponse:
+    def __init__(
+        self,
+        url,
+        data=mock_image_data,
+        content_type="image/png",
+        content_length=None,
+        status_code=200,
+    ):
+        self.url = url
+        self.chunks = [data]
+        self.status_code = status_code
+        if not content_length:
+            content_length = len(data)
+        self.headers = {"Content-Type": content_type, "Content-Length": content_length}
+
+    def iter_content(self, **kwargs):
+        return self.chunks
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        pass
+
+
+class PreviewImageLoaderTestCase(TestCase):
+    def setUp(self) -> None:
+        self.temp_folder = tempfile.TemporaryDirectory()
+        self.settings_override = self.settings(LD_PREVIEW_FOLDER=self.temp_folder.name)
+        self.settings_override.enable()
+        self.mock_load_website_metadata_patcher = mock.patch(
+            "bookmarks.services.website_loader.load_website_metadata"
+        )
+        self.mock_load_website_metadata = (
+            self.mock_load_website_metadata_patcher.start()
+        )
+        self.mock_load_website_metadata.return_value = mock.Mock(
+            preview_image="https://example.com/image.png"
+        )
+
+    def tearDown(self) -> None:
+        self.temp_folder.cleanup()
+        self.settings_override.disable()
+        self.mock_load_website_metadata_patcher.stop()
+
+    def create_mock_response(
+        self,
+        url="https://example.com/image.png",
+        icon_data=mock_image_data,
+        content_type="image/png",
+        content_length=len(mock_image_data),
+        status_code=200,
+    ):
+        mock_response = mock.Mock()
+        mock_response.raw = io.BytesIO(icon_data)
+        return MockStreamingResponse(
+            url, icon_data, content_type, content_length, status_code
+        )
+
+    def get_image_path(self, filename):
+        return Path(os.path.join(settings.LD_PREVIEW_FOLDER, filename))
+
+    def assertImageExists(self, filename, data):
+        self.assertTrue(self.get_image_path(filename).exists())
+        self.assertEqual(self.get_image_path(filename).read_bytes(), data)
+
+    def assertNoImageExists(self):
+        self.assertFalse(os.listdir(settings.LD_PREVIEW_FOLDER))
+
+    def test_load_preview_image(self):
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response()
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertIsNotNone(file)
+            self.assertImageExists(file, mock_image_data)
+
+    def test_load_preview_image_returns_none_if_no_preview_image_detected(self):
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response()
+            self.mock_load_website_metadata.return_value = mock.Mock(preview_image=None)
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertIsNone(file)
+            self.assertNoImageExists()
+
+    def test_load_preview_image_returns_none_for_invalid_status_code(self):
+        invalid_status_codes = [199, 300, 400, 500]
+
+        for status_code in invalid_status_codes:
+            with mock.patch("requests.get") as mock_get:
+                mock_get.return_value = self.create_mock_response(
+                    status_code=status_code
+                )
+
+                file = preview_image_loader.load_preview_image("https://example.com")
+
+                self.assertIsNone(file)
+                self.assertNoImageExists()
+
+    def test_load_preview_image_returns_none_if_content_length_exceeds_limit(self):
+        # exceeds max size
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response(
+                content_length=settings.LD_PREVIEW_MAX_SIZE + 1
+            )
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertIsNone(file)
+            self.assertNoImageExists()
+
+        # equals max size
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response(
+                content_length=settings.LD_PREVIEW_MAX_SIZE
+            )
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertIsNotNone(file)
+            self.assertImageExists(file, mock_image_data)
+
+    def test_load_preview_image_returns_none_for_invalid_content_type(self):
+        invalid_content_types = ["text/html", "application/json"]
+
+        for content_type in invalid_content_types:
+            with mock.patch("requests.get") as mock_get:
+                mock_get.return_value = self.create_mock_response(
+                    content_type=content_type
+                )
+
+                file = preview_image_loader.load_preview_image("https://example.com")
+
+                self.assertIsNone(file)
+                self.assertNoImageExists()
+
+        valid_content_types = ["image/png", "image/jpeg", "image/gif"]
+
+        for content_type in valid_content_types:
+            with mock.patch("requests.get") as mock_get:
+                mock_get.return_value = self.create_mock_response(
+                    content_type=content_type
+                )
+
+                file = preview_image_loader.load_preview_image("https://example.com")
+
+                self.assertIsNotNone(file)
+                self.assertImageExists(file, mock_image_data)
+
+    def test_load_preview_image_returns_none_if_download_exceeds_content_length(self):
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response(content_length=1)
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertIsNone(file)
+            self.assertNoImageExists()
+
+    def test_load_preview_image_creates_folder_if_not_exists(self):
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response()
+
+            folder = Path(settings.LD_PREVIEW_FOLDER)
+            folder.rmdir()
+
+            self.assertFalse(folder.exists())
+
+            preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertTrue(folder.exists())
+
+    def test_guess_file_extension(self):
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response(content_type="image/png")
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertImageExists(file, mock_image_data)
+            self.assertEqual("png", file.split(".")[-1])
+
+        with mock.patch("requests.get") as mock_get:
+            mock_get.return_value = self.create_mock_response(content_type="image/jpeg")
+
+            file = preview_image_loader.load_preview_image("https://example.com")
+
+            self.assertImageExists(file, mock_image_data)
+            self.assertEqual("jpg", file.split(".")[-1])

bookmarks/tests/test_settings_general_view.py

@@ -31,8 +31,10 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
             "enable_sharing": False,
             "enable_public_sharing": False,
             "enable_favicons": False,
+            "enable_preview_images": False,
             "enable_automatic_html_snapshots": True,
             "tag_search": UserProfile.TAG_SEARCH_STRICT,
+            "tag_grouping": UserProfile.TAG_GROUPING_ALPHABETICAL,
             "display_url": False,
             "display_view_bookmark_action": True,
             "display_edit_bookmark_action": True,
@@ -40,10 +42,29 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
             "display_remove_bookmark_action": True,
             "permanent_notes": False,
             "custom_css": "",
+            "auto_tagging_rules": "",
         }
 
         return {**form_data, **overrides}
 
+    def assertSuccessMessage(self, html, message: str, count=1):
+        self.assertInHTML(
+            f"""
+            <div class="toast toast-success mb-4">{ message }</div>
+        """,
+            html,
+            count=count,
+        )
+
+    def assertErrorMessage(self, html, message: str, count=1):
+        self.assertInHTML(
+            f"""
+            <div class="toast toast-error mb-4">{ message }</div>
+        """,
+            html,
+            count=count,
+        )
+
     def test_should_render_successfully(self):
         response = self.client.get(reverse("bookmarks:settings.general"))
 
@@ -70,15 +91,19 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
             "enable_sharing": True,
             "enable_public_sharing": True,
             "enable_favicons": True,
+            "enable_preview_images": True,
             "enable_automatic_html_snapshots": False,
             "tag_search": UserProfile.TAG_SEARCH_LAX,
+            "tag_grouping": UserProfile.TAG_GROUPING_DISABLED,
             "display_url": True,
             "display_view_bookmark_action": False,
             "display_edit_bookmark_action": False,
             "display_archive_bookmark_action": False,
             "display_remove_bookmark_action": False,
             "permanent_notes": True,
+            "default_mark_unread": True,
             "custom_css": "body { background-color: #000; }",
+            "auto_tagging_rules": "example.com tag",
         }
         response = self.client.post(reverse("bookmarks:settings.general"), form_data)
         html = response.content.decode()
@@ -112,11 +137,15 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
         self.assertEqual(
             self.user.profile.enable_favicons, form_data["enable_favicons"]
         )
+        self.assertEqual(
+            self.user.profile.enable_preview_images, form_data["enable_preview_images"]
+        )
         self.assertEqual(
             self.user.profile.enable_automatic_html_snapshots,
             form_data["enable_automatic_html_snapshots"],
         )
         self.assertEqual(self.user.profile.tag_search, form_data["tag_search"])
+        self.assertEqual(self.user.profile.tag_grouping, form_data["tag_grouping"])
         self.assertEqual(self.user.profile.display_url, form_data["display_url"])
         self.assertEqual(
             self.user.profile.display_view_bookmark_action,
@@ -137,13 +166,14 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
         self.assertEqual(
             self.user.profile.permanent_notes, form_data["permanent_notes"]
         )
-        self.assertEqual(self.user.profile.custom_css, form_data["custom_css"])
-        self.assertInHTML(
-            """
-                <p class="form-input-hint">Profile updated</p>
-            """,
-            html,
+        self.assertEqual(
+            self.user.profile.default_mark_unread, form_data["default_mark_unread"]
         )
+        self.assertEqual(self.user.profile.custom_css, form_data["custom_css"])
+        self.assertEqual(
+            self.user.profile.auto_tagging_rules, form_data["auto_tagging_rules"]
+        )
+        self.assertSuccessMessage(html, "Profile updated")
 
     def test_update_profile_should_not_be_called_without_respective_form_action(self):
         form_data = {
@@ -156,13 +186,7 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertEqual(response.status_code, 200)
         self.assertEqual(self.user.profile.theme, UserProfile.THEME_AUTO)
-        self.assertInHTML(
-            """
-                <p class="form-input-hint">Profile updated</p>
-            """,
-            html,
-            count=0,
-        )
+        self.assertSuccessMessage(html, "Profile updated", count=0)
 
     def test_enable_favicons_should_schedule_icon_update(self):
         with patch.object(
@@ -210,13 +234,8 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
             html = response.content.decode()
 
             mock_schedule_refresh_favicons.assert_called_once()
-            self.assertInHTML(
-                """
-                <p class="form-input-hint">
-                    Scheduled favicon update. This may take a while...
-                </p>
-            """,
-                html,
+            self.assertSuccessMessage(
+                html, "Scheduled favicon update. This may take a while..."
             )
 
     def test_refresh_favicons_should_not_be_called_without_respective_form_action(self):
@@ -230,14 +249,8 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
             html = response.content.decode()
 
             mock_schedule_refresh_favicons.assert_not_called()
-            self.assertInHTML(
-                """
-                <p class="form-input-hint">
-                    Scheduled favicon update. This may take a while...
-                </p>
-            """,
-                html,
-                count=0,
+            self.assertSuccessMessage(
+                html, "Scheduled favicon update. This may take a while...", count=0
             )
 
     def test_refresh_favicons_should_be_visible_when_favicons_enabled_in_profile(self):
@@ -291,6 +304,39 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
             count=0,
         )
 
+    def test_enable_preview_image_should_schedule_preview_update(self):
+        with patch.object(
+            tasks, "schedule_bookmarks_without_previews"
+        ) as mock_schedule_bookmarks_without_previews:
+            # Enabling favicons schedules update
+            form_data = self.create_profile_form_data(
+                {
+                    "update_profile": "",
+                    "enable_preview_images": True,
+                }
+            )
+            self.client.post(reverse("bookmarks:settings.general"), form_data)
+
+            mock_schedule_bookmarks_without_previews.assert_called_once_with(self.user)
+
+            # No update scheduled if favicons are already enabled
+            mock_schedule_bookmarks_without_previews.reset_mock()
+
+            self.client.post(reverse("bookmarks:settings.general"), form_data)
+
+            mock_schedule_bookmarks_without_previews.assert_not_called()
+
+            # No update scheduled when disabling favicons
+            form_data = self.create_profile_form_data(
+                {
+                    "enable_preview_images": False,
+                }
+            )
+
+            self.client.post(reverse("bookmarks:settings.general"), form_data)
+
+            mock_schedule_bookmarks_without_previews.assert_not_called()
+
     def test_automatic_html_snapshots_should_be_hidden_when_snapshots_not_supported(
         self,
     ):
@@ -365,3 +411,57 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
         with patch.object(requests, "get", return_value=latest_version_response_mock):
             version_info = get_version_info(random.random())
             self.assertEqual(version_info, app_version)
+
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_create_missing_html_snapshots(self):
+        with patch.object(
+            tasks, "create_missing_html_snapshots"
+        ) as mock_create_missing_html_snapshots:
+            mock_create_missing_html_snapshots.return_value = 5
+            form_data = {
+                "create_missing_html_snapshots": "",
+            }
+            response = self.client.post(
+                reverse("bookmarks:settings.general"), form_data
+            )
+            html = response.content.decode()
+
+            mock_create_missing_html_snapshots.assert_called_once()
+            self.assertSuccessMessage(
+                html, "Queued 5 missing snapshots. This may take a while..."
+            )
+
+    @override_settings(LD_ENABLE_SNAPSHOTS=True)
+    def test_create_missing_html_snapshots_no_missing_snapshots(self):
+        with patch.object(
+            tasks, "create_missing_html_snapshots"
+        ) as mock_create_missing_html_snapshots:
+            mock_create_missing_html_snapshots.return_value = 0
+            form_data = {
+                "create_missing_html_snapshots": "",
+            }
+            response = self.client.post(
+                reverse("bookmarks:settings.general"), form_data
+            )
+            html = response.content.decode()
+
+            mock_create_missing_html_snapshots.assert_called_once()
+            self.assertSuccessMessage(html, "No missing snapshots found.")
+
+    def test_create_missing_html_snapshots_should_not_be_called_without_respective_form_action(
+        self,
+    ):
+        with patch.object(
+            tasks, "create_missing_html_snapshots"
+        ) as mock_create_missing_html_snapshots:
+            mock_create_missing_html_snapshots.return_value = 5
+            form_data = {}
+            response = self.client.post(
+                reverse("bookmarks:settings.general"), form_data
+            )
+            html = response.content.decode()
+
+            mock_create_missing_html_snapshots.assert_not_called()
+            self.assertSuccessMessage(
+                html, "Queued 5 missing snapshots. This may take a while...", count=0
+            )

bookmarks/tests/test_settings_import_view.py

@@ -11,19 +11,27 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
         user = self.get_or_create_test_user()
         self.client.force_login(user)
 
-    def assertFormSuccessHint(self, response, text: str):
-        self.assertContains(response, '<div class="has-success">')
-        self.assertContains(response, text)
+    def assertSuccessMessage(self, response, message: str):
+        self.assertInHTML(
+            f"""
+            <div class="toast toast-success mb-4">{ message }</div>
+        """,
+            response.content.decode("utf-8"),
+        )
 
-    def assertNoFormSuccessHint(self, response):
-        self.assertNotContains(response, '<div class="has-success">')
+    def assertNoSuccessMessage(self, response):
+        self.assertNotContains(response, '<div class="toast toast-success mb-4">')
 
-    def assertFormErrorHint(self, response, text: str):
-        self.assertContains(response, '<div class="has-error">')
-        self.assertContains(response, text)
+    def assertErrorMessage(self, response, message: str):
+        self.assertInHTML(
+            f"""
+            <div class="toast toast-error mb-4">{ message }</div>
+        """,
+            response.content.decode("utf-8"),
+        )
 
-    def assertNoFormErrorHint(self, response):
-        self.assertNotContains(response, '<div class="has-error">')
+    def assertNoErrorMessage(self, response):
+        self.assertNotContains(response, '<div class="toast toast-error mb-4">')
 
     def test_should_import_successfully(self):
         with open(
@@ -36,10 +44,10 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
             )
 
             self.assertRedirects(response, reverse("bookmarks:settings.general"))
-            self.assertFormSuccessHint(
-                response, "3 bookmarks were successfully imported"
+            self.assertSuccessMessage(
+                response, "3 bookmarks were successfully imported."
             )
-            self.assertNoFormErrorHint(response)
+            self.assertNoErrorMessage(response)
 
     def test_should_check_authentication(self):
         self.client.logout()
@@ -53,8 +61,8 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
         response = self.client.post(reverse("bookmarks:settings.import"), follow=True)
 
         self.assertRedirects(response, reverse("bookmarks:settings.general"))
-        self.assertNoFormSuccessHint(response)
-        self.assertFormErrorHint(response, "Please select a file to import.")
+        self.assertNoSuccessMessage(response)
+        self.assertErrorMessage(response, "Please select a file to import.")
 
     @disable_logging
     def test_should_show_hint_if_import_raises_exception(self):
@@ -68,8 +76,8 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
             )
 
             self.assertRedirects(response, reverse("bookmarks:settings.general"))
-            self.assertNoFormSuccessHint(response)
-            self.assertFormErrorHint(
+            self.assertNoSuccessMessage(response)
+            self.assertErrorMessage(
                 response, "An error occurred during bookmark import."
             )
 
@@ -87,10 +95,13 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
             )
 
             self.assertRedirects(response, reverse("bookmarks:settings.general"))
-            self.assertFormSuccessHint(
-                response, "2 bookmarks were successfully imported"
+            self.assertSuccessMessage(
+                response, "2 bookmarks were successfully imported."
+            )
+            self.assertErrorMessage(
+                response,
+                "1 bookmarks could not be imported. Please check the logs for more details.",
             )
-            self.assertFormErrorHint(response, "1 bookmarks could not be imported")
 
     def test_should_respect_map_private_flag_option(self):
         with open(

bookmarks/tests/test_signals.py

@@ -1,17 +0,0 @@
-from unittest.mock import patch
-
-from django.test import TestCase
-
-from bookmarks.services import tasks
-from bookmarks.tests.helpers import BookmarkFactoryMixin
-
-
-class SignalsTestCase(TestCase, BookmarkFactoryMixin):
-    def test_login_should_schedule_snapshot_creation(self):
-        user = self.get_or_create_test_user()
-
-        with patch.object(
-            tasks, "schedule_bookmarks_without_snapshots"
-        ) as mock_schedule_bookmarks_without_snapshots:
-            self.client.force_login(user)
-            mock_schedule_bookmarks_without_snapshots.assert_called_once_with(user)

bookmarks/tests/test_singlefile_service.py

@@ -61,6 +61,10 @@ class SingleFileServiceTestCase(TestCase):
 
             expected_args = [
                 "single-file",
+                '--browser-arg="--headless=new"',
+                '--browser-arg="--user-data-dir=./chromium-profile"',
+                '--browser-arg="--no-sandbox"',
+                '--browser-arg="--load-extension=uBlock0.chromium"',
                 "http://example.com",
                 self.html_filepath + ".tmp",
             ]
@@ -79,6 +83,10 @@ class SingleFileServiceTestCase(TestCase):
 
             expected_args = [
                 "single-file",
+                '--browser-arg="--headless=new"',
+                '--browser-arg="--user-data-dir=./chromium-profile"',
+                '--browser-arg="--no-sandbox"',
+                '--browser-arg="--load-extension=uBlock0.chromium"',
                 "--some-option",
                 "some value",
                 "--another-option",
@@ -97,9 +105,9 @@ class SingleFileServiceTestCase(TestCase):
         with mock.patch("subprocess.Popen", return_value=mock_process):
             singlefile.create_snapshot("http://example.com", self.html_filepath)
 
-            mock_process.wait.assert_called_with(timeout=60)
+            mock_process.wait.assert_called_with(timeout=120)
 
-    @override_settings(LD_SINGLEFILE_TIMEOUT_SEC=120)
+    @override_settings(LD_SINGLEFILE_TIMEOUT_SEC=180)
     def test_create_snapshot_custom_timeout_setting(self):
         mock_process = mock.Mock()
         mock_process.wait.return_value = 0
@@ -108,4 +116,4 @@ class SingleFileServiceTestCase(TestCase):
         with mock.patch("subprocess.Popen", return_value=mock_process):
             singlefile.create_snapshot("http://example.com", self.html_filepath)
 
-            mock_process.wait.assert_called_with(timeout=120)
+            mock_process.wait.assert_called_with(timeout=180)

bookmarks/tests/test_tag_cloud_template.py

@@ -140,6 +140,43 @@ class TagCloudTemplateTest(TestCase, BookmarkFactoryMixin, HtmlTestMixin):
             ],
         )
 
+    def test_group_when_grouping_disabled(self):
+        profile = self.get_or_create_test_user().profile
+        profile.tag_grouping = UserProfile.TAG_GROUPING_DISABLED
+        profile.save()
+
+        tags = [
+            self.setup_tag(name="Cockatoo"),
+            self.setup_tag(name="Badger"),
+            self.setup_tag(name="Buffalo"),
+            self.setup_tag(name="Chihuahua"),
+            self.setup_tag(name="Alpaca"),
+            self.setup_tag(name="Coyote"),
+            self.setup_tag(name="Aardvark"),
+            self.setup_tag(name="Bumblebee"),
+            self.setup_tag(name="Armadillo"),
+        ]
+        self.setup_bookmark(tags=tags)
+
+        rendered_template = self.render_template()
+
+        self.assertTagGroups(
+            rendered_template,
+            [
+                [
+                    "Aardvark",
+                    "Alpaca",
+                    "Armadillo",
+                    "Badger",
+                    "Buffalo",
+                    "Bumblebee",
+                    "Chihuahua",
+                    "Cockatoo",
+                    "Coyote",
+                ],
+            ],
+        )
+
     def test_no_duplicate_tag_names(self):
         tags = [
             self.setup_tag(name="shared", user=self.setup_user(enable_sharing=True)),

bookmarks/tests/test_website_loader.py

@@ -29,7 +29,9 @@ class WebsiteLoaderTestCase(TestCase):
         # clear cached metadata before test run
         website_loader.load_website_metadata.cache_clear()
 
-    def render_html_document(self, title, description="", og_description=""):
+    def render_html_document(
+        self, title, description="", og_description="", og_image=""
+    ):
         meta_description = (
             f'<meta name="description" content="{description}">' if description else ""
         )
@@ -38,6 +40,9 @@ class WebsiteLoaderTestCase(TestCase):
             if og_description
             else ""
         )
+        meta_og_image = (
+            f'<meta property="og:image" content="{og_image}">' if og_image else ""
+        )
         return f"""
         <!DOCTYPE html>
         <html lang="en">
@@ -46,6 +51,7 @@ class WebsiteLoaderTestCase(TestCase):
             <title>{title}</title>
             {meta_description}
             {meta_og_description}
+            {meta_og_image}
         </head>
         <body></body>
         </html>
@@ -105,6 +111,7 @@ class WebsiteLoaderTestCase(TestCase):
             metadata = website_loader.load_website_metadata("https://example.com")
             self.assertEqual("test title", metadata.title)
             self.assertEqual("test description", metadata.description)
+            self.assertIsNone(metadata.preview_image)
 
     def test_load_website_metadata_trims_title_and_description(self):
         with mock.patch(
@@ -128,6 +135,44 @@ class WebsiteLoaderTestCase(TestCase):
             self.assertEqual("test title", metadata.title)
             self.assertEqual("test og description", metadata.description)
 
+    def test_load_website_metadata_using_og_image(self):
+        with mock.patch(
+            "bookmarks.services.website_loader.load_page"
+        ) as mock_load_page:
+            mock_load_page.return_value = self.render_html_document(
+                "test title", og_image="http://example.com/image.jpg"
+            )
+            metadata = website_loader.load_website_metadata("https://example.com")
+            self.assertEqual("http://example.com/image.jpg", metadata.preview_image)
+
+    def test_load_website_metadata_gets_absolute_og_image_path_when_path_starts_with_dots(
+        self,
+    ):
+        with mock.patch(
+            "bookmarks.services.website_loader.load_page"
+        ) as mock_load_page:
+            mock_load_page.return_value = self.render_html_document(
+                "test title", og_image="../image.jpg"
+            )
+            metadata = website_loader.load_website_metadata(
+                "https://example.com/a/b/page.html"
+            )
+            self.assertEqual("https://example.com/a/image.jpg", metadata.preview_image)
+
+    def test_load_website_metadata_gets_absolute_og_image_path_when_path_starts_with_slash(
+        self,
+    ):
+        with mock.patch(
+            "bookmarks.services.website_loader.load_page"
+        ) as mock_load_page:
+            mock_load_page.return_value = self.render_html_document(
+                "test title", og_image="/image.jpg"
+            )
+            metadata = website_loader.load_website_metadata(
+                "https://example.com/a/b/page.html"
+            )
+            self.assertEqual("https://example.com/image.jpg", metadata.preview_image)
+
     def test_load_website_metadata_prefers_description_over_og_description(self):
         with mock.patch(
             "bookmarks.services.website_loader.load_page"

bookmarks/urls.py

@@ -44,12 +44,22 @@ urlpatterns = [
         views.bookmarks.details_modal,
         name="details_modal",
     ),
+    path(
+        "bookmarks/<int:bookmark_id>/details_assets",
+        views.bookmarks.details_assets,
+        name="details_assets",
+    ),
     # Assets
     path(
         "assets/<int:asset_id>",
         views.assets.view,
         name="assets.view",
     ),
+    path(
+        "assets/<int:asset_id>/read",
+        views.assets.read,
+        name="assets.read",
+    ),
     # Partials
     path(
         "bookmarks/partials/bookmark-list/active",
@@ -61,6 +71,11 @@ urlpatterns = [
         partials.active_tag_cloud,
         name="partials.tag_cloud.active",
     ),
+    path(
+        "bookmarks/partials/tag-modal/active",
+        partials.active_tag_modal,
+        name="partials.tag_modal.active",
+    ),
     path(
         "bookmarks/partials/bookmark-list/archived",
         partials.archived_bookmark_list,
@@ -71,6 +86,11 @@ urlpatterns = [
         partials.archived_tag_cloud,
         name="partials.tag_cloud.archived",
     ),
+    path(
+        "bookmarks/partials/tag-modal/archived",
+        partials.archived_tag_modal,
+        name="partials.tag_modal.archived",
+    ),
     path(
         "bookmarks/partials/bookmark-list/shared",
         partials.shared_bookmark_list,
@@ -82,9 +102,9 @@ urlpatterns = [
         name="partials.tag_cloud.shared",
     ),
     path(
-        "bookmarks/partials/details-form/<int:bookmark_id>",
-        partials.details_form,
-        name="partials.details_form",
+        "bookmarks/partials/tag-modal/shared",
+        partials.shared_tag_modal,
+        name="partials.tag_modal.shared",
     ),
     # Settings
     path("settings", views.settings.general, name="settings.index"),

bookmarks/views/assets.py

@@ -6,11 +6,12 @@ from django.http import (
     HttpResponse,
     Http404,
 )
+from django.shortcuts import render
 
 from bookmarks.models import BookmarkAsset
 
 
-def view(request, asset_id: int):
+def _access_asset(request, asset_id: int):
     try:
         asset = BookmarkAsset.objects.get(pk=asset_id)
     except BookmarkAsset.DoesNotExist:
@@ -28,6 +29,10 @@ def view(request, asset_id: int):
     if not is_owner and not is_shared and not is_public_shared:
         raise Http404("Bookmark does not exist")
 
+    return asset
+
+
+def _get_asset_content(asset):
     filepath = os.path.join(settings.LD_ASSET_FOLDER, asset.file)
 
     if not os.path.exists(filepath):
@@ -40,4 +45,25 @@ def view(request, asset_id: int):
         with open(filepath, "rb") as f:
             content = f.read()
 
+    return content
+
+
+def view(request, asset_id: int):
+    asset = _access_asset(request, asset_id)
+    content = _get_asset_content(asset)
+
     return HttpResponse(content, content_type=asset.content_type)
+
+
+def read(request, asset_id: int):
+    asset = _access_asset(request, asset_id)
+    content = _get_asset_content(asset)
+    content = content.decode("utf-8")
+
+    return render(
+        request,
+        "bookmarks/read.html",
+        {
+            "content": content,
+        },
+    )

bookmarks/views/bookmarks.py

@@ -34,7 +34,7 @@ from bookmarks.services.bookmarks import (
     share_bookmarks,
     unshare_bookmarks,
 )
-from bookmarks.services import tasks
+from bookmarks.services import bookmarks as bookmark_actions, tasks
 from bookmarks.utils import get_safe_return_url
 from bookmarks.views.partials import contexts
 
@@ -145,6 +145,11 @@ def _details(request, bookmark_id: int, template: str):
             asset.delete()
         if "create_snapshot" in request.POST:
             tasks.create_html_snapshot(bookmark)
+        if "upload_asset" in request.POST:
+            file = request.FILES.get("upload_asset_file")
+            if not file:
+                return HttpResponseBadRequest("No file uploaded")
+            bookmark_actions.upload_asset(bookmark, file)
         else:
             bookmark.is_archived = request.POST.get("is_archived") == "on"
             bookmark.unread = request.POST.get("unread") == "on"
@@ -172,6 +177,10 @@ def details_modal(request, bookmark_id: int):
     return _details(request, bookmark_id, "bookmarks/details_modal.html")
 
 
+def details_assets(request, bookmark_id: int):
+    return _details(request, bookmark_id, "bookmarks/details/assets.html")
+
+
 def convert_tag_string(tag_string: str):
     # Tag strings coming from inputs are space-separated, however services.bookmarks functions expect comma-separated
     # strings
@@ -184,6 +193,7 @@ def new(request):
     initial_title = request.GET.get("title")
     initial_description = request.GET.get("description")
     initial_auto_close = "auto_close" in request.GET
+    initial_mark_unread = request.user.profile.default_mark_unread
 
     if request.method == "POST":
         form = BookmarkForm(request.POST)
@@ -206,6 +216,8 @@ def new(request):
             form.initial["description"] = initial_description
         if initial_auto_close:
             form.initial["auto_close"] = "true"
+        if initial_mark_unread:
+            form.initial["unread"] = "true"
 
     context = {
         "form": form,

bookmarks/views/partials/__init__.py

@@ -1,8 +1,6 @@
 from django.contrib.auth.decorators import login_required
-from django.http import Http404
 from django.shortcuts import render
 
-from bookmarks.models import Bookmark
 from bookmarks.views.partials import contexts
 
 
@@ -24,6 +22,13 @@ def active_tag_cloud(request):
     return render(request, "bookmarks/tag_cloud.html", {"tag_cloud": tag_cloud_context})
 
 
+@login_required
+def active_tag_modal(request):
+    tag_cloud_context = contexts.ActiveTagCloudContext(request)
+
+    return render(request, "bookmarks/tag_modal.html", {"tag_cloud": tag_cloud_context})
+
+
 @login_required
 def archived_bookmark_list(request):
     bookmark_list_context = contexts.ArchivedBookmarkListContext(request)
@@ -43,6 +48,12 @@ def archived_tag_cloud(request):
 
 
 @login_required
+def archived_tag_modal(request):
+    tag_cloud_context = contexts.ArchivedTagCloudContext(request)
+
+    return render(request, "bookmarks/tag_modal.html", {"tag_cloud": tag_cloud_context})
+
+
 def shared_bookmark_list(request):
     bookmark_list_context = contexts.SharedBookmarkListContext(request)
 
@@ -53,20 +64,13 @@ def shared_bookmark_list(request):
     )
 
 
-@login_required
 def shared_tag_cloud(request):
     tag_cloud_context = contexts.SharedTagCloudContext(request)
 
     return render(request, "bookmarks/tag_cloud.html", {"tag_cloud": tag_cloud_context})
 
 
-@login_required
-def details_form(request, bookmark_id: int):
-    try:
-        bookmark = Bookmark.objects.get(pk=bookmark_id, owner=request.user)
-    except Bookmark.DoesNotExist:
-        raise Http404("Bookmark does not exist")
+def shared_tag_modal(request):
+    tag_cloud_context = contexts.SharedTagCloudContext(request)
 
-    details_context = contexts.BookmarkDetailsContext(request, bookmark)
-
-    return render(request, "bookmarks/details/form.html", {"details": details_context})
+    return render(request, "bookmarks/tag_modal.html", {"tag_cloud": tag_cloud_context})

bookmarks/views/partials/contexts.py

@@ -1,12 +1,12 @@
+import re
 import urllib.parse
 from typing import Set, List
-import re
 
+from django.conf import settings
 from django.core.handlers.wsgi import WSGIRequest
 from django.core.paginator import Paginator
 from django.db import models
 from django.urls import reverse
-from django.conf import settings
 
 from bookmarks import queries
 from bookmarks import utils
@@ -18,11 +18,119 @@ from bookmarks.models import (
     UserProfile,
     Tag,
 )
+from bookmarks.services.wayback import generate_fallback_webarchive_url
 
 DEFAULT_PAGE_SIZE = 30
 CJK_RE = re.compile(r"[\u4e00-\u9fff]+")
 
 
+class RequestContext:
+    index_view = "bookmarks:index"
+    action_view = "bookmarks:index.action"
+    bookmark_list_partial_view = "bookmarks:partials.bookmark_list.active"
+    tag_cloud_partial_view = "bookmarks:partials.tag_cloud.active"
+    tag_modal_partial_view = "bookmarks:partials.tag_modal.active"
+
+    def __init__(self, request: WSGIRequest):
+        self.request = request
+        self.index_url = reverse(self.index_view)
+        self.action_url = reverse(self.action_view)
+        self.bookmark_list_partial_url = reverse(self.bookmark_list_partial_view)
+        self.tag_cloud_partial_url = reverse(self.tag_cloud_partial_view)
+        self.tag_modal_partial_url = reverse(self.tag_modal_partial_view)
+        self.query_params = request.GET.copy()
+        self.query_params.pop("details", None)
+
+    def get_url(self, view_url: str, add: dict = None, remove: dict = None) -> str:
+        query_params = self.query_params.copy()
+        if add:
+            query_params.update(add)
+        if remove:
+            for key in remove:
+                query_params.pop(key, None)
+        encoded_params = query_params.urlencode()
+        return view_url + "?" + encoded_params if encoded_params else view_url
+
+    def index(self) -> str:
+        return self.get_url(self.index_url)
+
+    def action(self, return_url: str) -> str:
+        return self.get_url(self.action_url, add={"return_url": return_url})
+
+    def bookmark_list_partial(self) -> str:
+        return self.get_url(self.bookmark_list_partial_url)
+
+    def tag_cloud_partial(self) -> str:
+        return self.get_url(self.tag_cloud_partial_url)
+
+    def tag_modal_partial(self) -> str:
+        return self.get_url(self.tag_modal_partial_url)
+
+    def get_bookmark_query_set(self, search: BookmarkSearch):
+        raise Exception("Must be implemented by subclass")
+
+    def get_tag_query_set(self, search: BookmarkSearch):
+        raise Exception("Must be implemented by subclass")
+
+
+class ActiveBookmarksContext(RequestContext):
+    index_view = "bookmarks:index"
+    action_view = "bookmarks:index.action"
+    bookmark_list_partial_view = "bookmarks:partials.bookmark_list.active"
+    tag_cloud_partial_view = "bookmarks:partials.tag_cloud.active"
+    tag_modal_partial_view = "bookmarks:partials.tag_modal.active"
+
+    def get_bookmark_query_set(self, search: BookmarkSearch):
+        return queries.query_bookmarks(
+            self.request.user, self.request.user_profile, search
+        )
+
+    def get_tag_query_set(self, search: BookmarkSearch):
+        return queries.query_bookmark_tags(
+            self.request.user, self.request.user_profile, search
+        )
+
+
+class ArchivedBookmarksContext(RequestContext):
+    index_view = "bookmarks:archived"
+    action_view = "bookmarks:archived.action"
+    bookmark_list_partial_view = "bookmarks:partials.bookmark_list.archived"
+    tag_cloud_partial_view = "bookmarks:partials.tag_cloud.archived"
+    tag_modal_partial_view = "bookmarks:partials.tag_modal.archived"
+
+    def get_bookmark_query_set(self, search: BookmarkSearch):
+        return queries.query_archived_bookmarks(
+            self.request.user, self.request.user_profile, search
+        )
+
+    def get_tag_query_set(self, search: BookmarkSearch):
+        return queries.query_archived_bookmark_tags(
+            self.request.user, self.request.user_profile, search
+        )
+
+
+class SharedBookmarksContext(RequestContext):
+    index_view = "bookmarks:shared"
+    action_view = "bookmarks:shared.action"
+    bookmark_list_partial_view = "bookmarks:partials.bookmark_list.shared"
+    tag_cloud_partial_view = "bookmarks:partials.tag_cloud.shared"
+    tag_modal_partial_view = "bookmarks:partials.tag_modal.shared"
+
+    def get_bookmark_query_set(self, search: BookmarkSearch):
+        user = User.objects.filter(username=search.user).first()
+        public_only = not self.request.user.is_authenticated
+        return queries.query_shared_bookmarks(
+            user, self.request.user_profile, search, public_only
+        )
+
+    def get_tag_query_set(self, search: BookmarkSearch):
+        user = User.objects.filter(username=search.user).first()
+        public_only = not self.request.user.is_authenticated
+        return queries.query_shared_bookmark_tags(
+            user, self.request.user_profile, search, public_only
+        )
+
+
 class BookmarkItem:
     def __init__(self, bookmark: Bookmark, user: User, profile: UserProfile) -> None:
         self.bookmark = bookmark
@@ -37,7 +145,12 @@ class BookmarkItem:
         self.notes = bookmark.notes
         self.tag_names = bookmark.tag_names
         self.web_archive_snapshot_url = bookmark.web_archive_snapshot_url
+        if not self.web_archive_snapshot_url:
+            self.web_archive_snapshot_url = generate_fallback_webarchive_url(
+                bookmark.url, bookmark.date_added
+            )
         self.favicon_file = bookmark.favicon_file
+        self.preview_image_file = bookmark.preview_image_file
         self.is_archived = bookmark.is_archived
         self.unread = bookmark.unread
         self.owner = bookmark.owner
@@ -67,7 +180,10 @@ class BookmarkItem:
 
 
 class BookmarkListContext:
+    request_context = RequestContext
+
     def __init__(self, request: WSGIRequest) -> None:
+        request_context = self.request_context(request)
         user = request.user
         user_profile = request.user_profile
 
@@ -76,7 +192,7 @@ class BookmarkListContext:
             self.request.GET, user_profile.search_preferences
         )
 
-        query_set = self.get_bookmark_query_set()
+        query_set = request_context.get_bookmark_query_set(self.search)
         page_number = request.GET.get("page")
         paginator = Paginator(query_set, DEFAULT_PAGE_SIZE)
         bookmarks_page = paginator.get_page(page_number)
@@ -86,16 +202,15 @@ class BookmarkListContext:
         self.items = [
             BookmarkItem(bookmark, user, user_profile) for bookmark in bookmarks_page
         ]
-
         self.is_empty = paginator.count == 0
         self.bookmarks_page = bookmarks_page
         self.bookmarks_total = paginator.count
-        self.return_url = self.generate_return_url(
-            self.search, self.get_base_url(), page_number
-        )
-        self.action_url = self.generate_action_url(
-            self.search, self.get_base_action_url(), self.return_url
-        )
+
+        self.return_url = request_context.index()
+        self.action_url = request_context.action(return_url=self.return_url)
+        self.refresh_url = request_context.bookmark_list_partial()
+        self.tag_modal_url = request_context.tag_modal_partial()
+
         self.link_target = user_profile.bookmark_link_target
         self.date_display = user_profile.bookmark_date_display
         self.description_display = user_profile.bookmark_description_display
@@ -106,6 +221,7 @@ class BookmarkListContext:
         self.show_archive_action = user_profile.display_archive_bookmark_action
         self.show_remove_action = user_profile.display_remove_bookmark_action
         self.show_favicons = user_profile.enable_favicons
+        self.show_preview_images = user_profile.enable_preview_images
         self.show_notes = user_profile.permanent_notes
 
     @staticmethod
@@ -131,55 +247,17 @@ class BookmarkListContext:
             else base_action_url + "?" + query_string
         )
 
-    def get_base_url(self):
-        raise Exception("Must be implemented by subclass")
-
-    def get_base_action_url(self):
-        raise Exception("Must be implemented by subclass")
-
-    def get_bookmark_query_set(self):
-        raise Exception("Must be implemented by subclass")
-
 
 class ActiveBookmarkListContext(BookmarkListContext):
-    def get_base_url(self):
-        return reverse("bookmarks:index")
-
-    def get_base_action_url(self):
-        return reverse("bookmarks:index.action")
-
-    def get_bookmark_query_set(self):
-        return queries.query_bookmarks(
-            self.request.user, self.request.user_profile, self.search
-        )
+    request_context = ActiveBookmarksContext
 
 
 class ArchivedBookmarkListContext(BookmarkListContext):
-    def get_base_url(self):
-        return reverse("bookmarks:archived")
-
-    def get_base_action_url(self):
-        return reverse("bookmarks:archived.action")
-
-    def get_bookmark_query_set(self):
-        return queries.query_archived_bookmarks(
-            self.request.user, self.request.user_profile, self.search
-        )
+    request_context = ArchivedBookmarksContext
 
 
 class SharedBookmarkListContext(BookmarkListContext):
-    def get_base_url(self):
-        return reverse("bookmarks:shared")
-
-    def get_base_action_url(self):
-        return reverse("bookmarks:shared.action")
-
-    def get_bookmark_query_set(self):
-        user = User.objects.filter(username=self.search.user).first()
-        public_only = not self.request.user.is_authenticated
-        return queries.query_shared_bookmarks(
-            user, self.request.user_profile, self.search, public_only
-        )
+    request_context = SharedBookmarksContext
 
 
 class TagGroup:
@@ -191,7 +269,16 @@ class TagGroup:
         return f"<{self.char} TagGroup>"
 
     @staticmethod
-    def create_tag_groups(tags: Set[Tag]):
+    def create_tag_groups(mode: str, tags: Set[Tag]):
+        if mode == UserProfile.TAG_GROUPING_ALPHABETICAL:
+            return TagGroup._create_tag_groups_alphabetical(tags)
+        elif mode == UserProfile.TAG_GROUPING_DISABLED:
+            return TagGroup._create_tag_groups_disabled(tags)
+        else:
+            raise ValueError(f"{mode} is not a valid tag grouping mode")
+
+    @staticmethod
+    def _create_tag_groups_alphabetical(tags: Set[Tag]):
         # Ensure groups, as well as tags within groups, are ordered alphabetically
         sorted_tags = sorted(tags, key=lambda x: str.lower(x.name))
         group = None
@@ -216,9 +303,24 @@ class TagGroup:
             groups.append(cjk_group)
         return groups
 
+    @staticmethod
+    def _create_tag_groups_disabled(tags: Set[Tag]):
+        if len(tags) == 0:
+            return []
+
+        sorted_tags = sorted(tags, key=lambda x: str.lower(x.name))
+        group = TagGroup("Ungrouped")
+        for tag in sorted_tags:
+            group.tags.append(tag)
+
+        return [group]
+
 
 class TagCloudContext:
+    request_context = RequestContext
+
     def __init__(self, request: WSGIRequest) -> None:
+        request_context = self.request_context(request)
         user_profile = request.user_profile
 
         self.request = request
@@ -226,7 +328,7 @@ class TagCloudContext:
             self.request.GET, user_profile.search_preferences
         )
 
-        query_set = self.get_tag_query_set()
+        query_set = request_context.get_tag_query_set(self.search)
         tags = list(query_set)
         selected_tags = self.get_selected_tags(tags)
         unique_tags = utils.unique(tags, key=lambda x: str.lower(x.name))
@@ -235,15 +337,14 @@ class TagCloudContext:
         )
         has_selected_tags = len(unique_selected_tags) > 0
         unselected_tags = set(unique_tags).symmetric_difference(unique_selected_tags)
-        groups = TagGroup.create_tag_groups(unselected_tags)
+        groups = TagGroup.create_tag_groups(user_profile.tag_grouping, unselected_tags)
 
         self.tags = unique_tags
         self.groups = groups
         self.selected_tags = unique_selected_tags
         self.has_selected_tags = has_selected_tags
 
-    def get_tag_query_set(self):
-        raise Exception("Must be implemented by subclass")
+        self.refresh_url = request_context.tag_cloud_partial()
 
     def get_selected_tags(self, tags: List[Tag]):
         parsed_query = queries.parse_query_string(self.search.q)
@@ -256,26 +357,15 @@ class TagCloudContext:
 
 
 class ActiveTagCloudContext(TagCloudContext):
-    def get_tag_query_set(self):
-        return queries.query_bookmark_tags(
-            self.request.user, self.request.user_profile, self.search
-        )
+    request_context = ActiveBookmarksContext
 
 
 class ArchivedTagCloudContext(TagCloudContext):
-    def get_tag_query_set(self):
-        return queries.query_archived_bookmark_tags(
-            self.request.user, self.request.user_profile, self.search
-        )
+    request_context = ArchivedBookmarksContext
 
 
 class SharedTagCloudContext(TagCloudContext):
-    def get_tag_query_set(self):
-        user = User.objects.filter(username=self.search.user).first()
-        public_only = not self.request.user.is_authenticated
-        return queries.query_shared_bookmark_tags(
-            user, self.request.user_profile, self.search, public_only
-        )
+    request_context = SharedBookmarksContext
 
 
 class BookmarkAssetItem:
@@ -284,6 +374,7 @@ class BookmarkAssetItem:
 
         self.id = asset.id
         self.display_name = asset.display_name
+        self.asset_type = asset.asset_type
         self.content_type = asset.content_type
         self.file = asset.file
         self.file_size = asset.file_size
@@ -321,10 +412,29 @@ class BookmarkDetailsContext:
         self.profile = request.user_profile
         self.is_editable = bookmark.owner == user
         self.sharing_enabled = user_profile.enable_sharing
+        self.preview_image_enabled = user_profile.enable_preview_images
         self.show_link_icons = user_profile.enable_favicons and bookmark.favicon_file
         # For now hide files section if snapshots are not supported
         self.show_files = settings.LD_ENABLE_SNAPSHOTS
 
+        self.web_archive_snapshot_url = bookmark.web_archive_snapshot_url
+        if not self.web_archive_snapshot_url:
+            self.web_archive_snapshot_url = generate_fallback_webarchive_url(
+                bookmark.url, bookmark.date_added
+            )
+
         self.assets = [
             BookmarkAssetItem(asset) for asset in bookmark.bookmarkasset_set.all()
         ]
+        self.has_pending_assets = any(
+            asset.status == BookmarkAsset.STATUS_PENDING for asset in self.assets
+        )
+        self.latest_snapshot = next(
+            (
+                asset
+                for asset in self.assets
+                if asset.asset.asset_type == BookmarkAsset.TYPE_SNAPSHOT
+                and asset.status == BookmarkAsset.STATUS_COMPLETE
+            ),
+            None,
+        )

bookmarks/views/settings.py

@@ -25,12 +25,10 @@ def general(request):
     profile_form = None
     enable_refresh_favicons = django_settings.LD_ENABLE_REFRESH_FAVICONS
     has_snapshot_support = django_settings.LD_ENABLE_SNAPSHOTS
-    update_profile_success_message = None
-    refresh_favicons_success_message = None
-    import_success_message = _find_message_with_tag(
+    success_message = _find_message_with_tag(
         messages.get_messages(request), "bookmark_import_success"
     )
-    import_errors_message = _find_message_with_tag(
+    error_message = _find_message_with_tag(
         messages.get_messages(request), "bookmark_import_errors"
     )
     version_info = get_version_info(get_ttl_hash())
@@ -38,12 +36,18 @@ def general(request):
     if request.method == "POST":
         if "update_profile" in request.POST:
             profile_form = update_profile(request)
-            update_profile_success_message = "Profile updated"
+            success_message = "Profile updated"
         if "refresh_favicons" in request.POST:
             tasks.schedule_refresh_favicons(request.user)
-            refresh_favicons_success_message = (
-                "Scheduled favicon update. This may take a while..."
-            )
+            success_message = "Scheduled favicon update. This may take a while..."
+        if "create_missing_html_snapshots" in request.POST:
+            count = tasks.create_missing_html_snapshots(request.user)
+            if count > 0:
+                success_message = (
+                    f"Queued {count} missing snapshots. This may take a while..."
+                )
+            else:
+                success_message = "No missing snapshots found."
 
     if not profile_form:
         profile_form = UserProfileForm(instance=request.user_profile)
@@ -55,10 +59,8 @@ def general(request):
             "form": profile_form,
             "enable_refresh_favicons": enable_refresh_favicons,
             "has_snapshot_support": has_snapshot_support,
-            "update_profile_success_message": update_profile_success_message,
-            "refresh_favicons_success_message": refresh_favicons_success_message,
-            "import_success_message": import_success_message,
-            "import_errors_message": import_errors_message,
+            "success_message": success_message,
+            "error_message": error_message,
             "version_info": version_info,
         },
     )
@@ -68,11 +70,16 @@ def update_profile(request):
     user = request.user
     profile = user.profile
     favicons_were_enabled = profile.enable_favicons
+    previews_were_enabled = profile.enable_preview_images
     form = UserProfileForm(request.POST, instance=profile)
     if form.is_valid():
         form.save()
+        # Load missing favicons if the feature was just enabled
         if profile.enable_favicons and not favicons_were_enabled:
             tasks.schedule_bookmarks_without_favicons(request.user)
+        # Load missing preview images if the feature was just enabled
+        if profile.enable_preview_images and not previews_were_enabled:
+            tasks.schedule_bookmarks_without_previews(request.user)
     return form
 
 

bootstrap.sh

@@ -7,6 +7,8 @@ LD_SERVER_PORT="${LD_SERVER_PORT:-9090}"
 mkdir -p data
 # Create favicon folder if it does not exist
 mkdir -p data/favicons
+# Create previews folder if it does not exist
+mkdir -p data/previews
 # Create assets folder if it does not exist
 mkdir -p data/assets
 
@@ -21,7 +23,7 @@ python manage.py create_initial_superuser
 # Migrate legacy background tasks to Huey
 python manage.py migrate_tasks
 
-# Ensure the DB folder is owned by the right user
+# Ensure folders are owned by the right user
 chown -R www-data: /etc/linkding/data
 
 # Start background task processor using supervisord, unless explicitly disabled

docker-compose.yml

@@ -1,5 +1,3 @@
-version: '3'
-
 services:
   linkding:
     container_name: "${LD_CONTAINER_NAME:-linkding}"
@@ -10,4 +8,4 @@ services:
       - "${LD_HOST_DATA_DIR:-./data}:/etc/linkding/data"
     env_file:
       - .env
-    restart: unless-stopped
+    restart: unless-stopped

docker/alpine.Dockerfile

@@ -69,6 +69,7 @@ RUN wget https://www.sqlite.org/${SQLITE_RELEASE_YEAR}/sqlite-amalgamation-${SQL
 
 
 FROM python:3.11.8-alpine3.19 AS linkding
+LABEL org.opencontainers.image.source="https://github.com/sissbruecker/linkding"
 # install runtime dependencies
 RUN apk update && apk add bash curl icu libpq mailcap libssl3
 # create www-data user and group
@@ -99,10 +100,31 @@ CMD curl -f http://localhost:${LD_SERVER_PORT:-9090}/${LD_CONTEXT_PATH}health ||
 CMD ["./bootstrap.sh"]
 
 
+FROM node:18-alpine AS ublock-build
+WORKDIR /etc/linkding
+# Install necessary tools
+RUN apk add --no-cache curl jq unzip
+# Fetch the latest release tag
+# Download the library
+# Unzip the library
+RUN TAG=$(curl -sL https://api.github.com/repos/gorhill/uBlock/releases/latest | jq -r '.tag_name') && \
+    DOWNLOAD_URL=https://github.com/gorhill/uBlock/releases/download/$TAG/uBlock0_$TAG.chromium.zip && \
+    curl -L -o uBlock0.zip $DOWNLOAD_URL && \
+    unzip uBlock0.zip
+# Patch assets.json to enable easylist-cookies by default
+RUN curl -L -o ./uBlock0.chromium/assets/thirdparties/easylist/easylist-cookies.txt https://ublockorigin.github.io/uAssets/thirdparties/easylist-cookies.txt
+RUN jq '."assets.json" |= del(.cdnURLs) | ."assets.json".contentURL = ["assets/assets.json"] | ."fanboy-cookiemonster" |= del(.off) | ."fanboy-cookiemonster".contentURL += ["assets/thirdparties/easylist/easylist-cookies.txt"]' ./uBlock0.chromium/assets/assets.json > temp.json && \
+    mv temp.json ./uBlock0.chromium/assets/assets.json
+
+
 FROM linkding AS linkding-plus
 # install node, chromium
 RUN apk update && apk add nodejs npm chromium
 # install single-file from fork for now, which contains several hotfixes
-RUN npm install -g https://github.com/sissbruecker/single-file-cli/tarball/f3730995a52f27d5041a1ad9e7528af4b6b4cf4b
+RUN npm install -g https://github.com/sissbruecker/single-file-cli/tarball/4c54b3bc704cfb3e96cec2d24854caca3df0b3b6
+# copy uBlock0
+COPY --from=ublock-build /etc/linkding/uBlock0.chromium uBlock0.chromium/
+# create chromium profile folder for user running background tasks
+RUN mkdir -p chromium-profile && chown -R www-data:www-data chromium-profile
 # enable snapshot support
 ENV LD_ENABLE_SNAPSHOTS=True

docker/default.Dockerfile

@@ -71,6 +71,7 @@ RUN wget https://www.sqlite.org/${SQLITE_RELEASE_YEAR}/sqlite-amalgamation-${SQL
 
 
 FROM python:3.11.8-slim-bookworm as linkding
+LABEL org.opencontainers.image.source="https://github.com/sissbruecker/linkding"
 RUN apt-get update && apt-get -y install mime-support libpq-dev libicu-dev libssl3 curl
 WORKDIR /etc/linkding
 # copy prod dependencies
@@ -96,6 +97,24 @@ CMD curl -f http://localhost:${LD_SERVER_PORT:-9090}/${LD_CONTEXT_PATH}health ||
 
 CMD ["./bootstrap.sh"]
 
+
+FROM node:18-alpine AS ublock-build
+WORKDIR /etc/linkding
+# Install necessary tools
+RUN apk add --no-cache curl jq unzip
+# Fetch the latest release tag
+# Download the library
+# Unzip the library
+RUN TAG=$(curl -sL https://api.github.com/repos/gorhill/uBlock/releases/latest | jq -r '.tag_name') && \
+    DOWNLOAD_URL=https://github.com/gorhill/uBlock/releases/download/$TAG/uBlock0_$TAG.chromium.zip && \
+    curl -L -o uBlock0.zip $DOWNLOAD_URL && \
+    unzip uBlock0.zip
+# Patch assets.json to enable easylist-cookies by default
+RUN curl -L -o ./uBlock0.chromium/assets/thirdparties/easylist/easylist-cookies.txt https://ublockorigin.github.io/uAssets/thirdparties/easylist-cookies.txt
+RUN jq '."assets.json" |= del(.cdnURLs) | ."assets.json".contentURL = ["assets/assets.json"] | ."fanboy-cookiemonster" |= del(.off) | ."fanboy-cookiemonster".contentURL += ["assets/thirdparties/easylist/easylist-cookies.txt"]' ./uBlock0.chromium/assets/assets.json > temp.json && \
+    mv temp.json ./uBlock0.chromium/assets/assets.json
+
+
 FROM linkding AS linkding-plus
 # install chromium
 RUN apt-get update && apt-get -y install chromium
@@ -106,6 +125,10 @@ RUN apt-get install -y gnupg2 apt-transport-https ca-certificates && \
     echo "deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
     apt-get update && apt-get install -y nodejs
 # install single-file from fork for now, which contains several hotfixes
-RUN npm install -g https://github.com/sissbruecker/single-file-cli/tarball/f3730995a52f27d5041a1ad9e7528af4b6b4cf4b
+RUN npm install -g https://github.com/sissbruecker/single-file-cli/tarball/4c54b3bc704cfb3e96cec2d24854caca3df0b3b6
+# create chromium profile folder for user running background tasks
+RUN mkdir -p chromium-profile && chown -R www-data:www-data chromium-profile
+# copy uBlock0
+COPY --from=ublock-build /etc/linkding/uBlock0.chromium uBlock0.chromium/
 # enable snapshot support
 ENV LD_ENABLE_SNAPSHOTS=True

docs/API.md

@@ -49,6 +49,8 @@ Example response:
       "website_title": "Website title",
       "website_description": "Website description",
       "web_archive_snapshot_url": "https://web.archive.org/web/20200926094623/https://example.com",
+      "favicon_url": "http://127.0.0.1:8000/static/https_example_com.png",
+      "preview_image_url": "http://127.0.0.1:8000/static/0ac5c53db923727765216a3a58e70522.jpg",
       "is_archived": false,
       "unread": false,
       "shared": false,

docs/backup.md

@@ -8,12 +8,13 @@ The data folder contains the following contents that are relevant for backups:
 - `db.sqlite3` - the SQLite database
 - `assets` - folder that contains HTML snapshots of bookmarks
 - `favicons` - folder that contains downloaded favicons
+- `previews` - folder that contains downloaded preview images
 
 The following sections explain how to back up the individual contents.
 
 ## Full backup
 
-linkding provides a CLI command to create a full backup of the data folder. This creates a zip file that contains backups of the database, assets, and favicons.
+linkding provides a CLI command to create a full backup of the data folder. This creates a zip file that contains backups of the database, assets, favicons, and preview images.
 
 > [!NOTE]
 > This method assumes that you are using the default SQLite database.
@@ -90,7 +91,7 @@ This is the least technical option to back up bookmarks, but has several limitat
 - It only exports your own bookmarks, not those of other users.
 - It does not export URLs of snapshots on the Internet Archive Wayback machine.
 - It does not export HTML snapshots of bookmarks. Even if you backup and restore the assets folder, the bookmarks will not be linked to the snapshots anymore.
-- It does not export favicons.
+- It does not export favicons or preview images.
 
 Only use this method if you are fine with the above limitations.
 

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "linkding",
-  "version": "1.25.0",
+  "version": "1.30.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "linkding",
-      "version": "1.25.0",
+      "version": "1.30.0",
       "license": "MIT",
       "dependencies": {
         "@rollup/plugin-node-resolve": "^15.2.3",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "linkding",
-  "version": "1.28.0",
+  "version": "1.31.1",
   "description": "",
   "main": "index.js",
   "scripts": {
@@ -28,5 +28,6 @@
   },
   "devDependencies": {
     "prettier": "^3.0.2"
-  }
+  },
+  "web-types": "./web-types.json"
 }

requirements.dev.in

@@ -6,3 +6,4 @@ libsass
 playwright
 pytest
 pytest-django
+pytest-xdist

requirements.dev.txt

@@ -12,7 +12,7 @@ click==8.1.7
     # via black
 coverage==7.4.1
     # via -r requirements.dev.in
-django==5.0.3
+django==5.0.8
     # via
     #   django-appconf
     #   django-debug-toolbar
@@ -22,6 +22,8 @@ django-compressor==4.4
     # via -r requirements.dev.in
 django-debug-toolbar==4.2.0
     # via -r requirements.dev.in
+execnet==2.1.1
+    # via pytest-xdist
 greenlet==3.0.3
     # via playwright
 iniconfig==2.0.0
@@ -48,13 +50,16 @@ pytest==8.0.0
     # via
     #   -r requirements.dev.in
     #   pytest-django
+    #   pytest-xdist
 pytest-django==4.7.0
     # via -r requirements.dev.in
+pytest-xdist==3.6.1
+    # via -r requirements.dev.in
 rcssmin==1.1.1
     # via django-compressor
 rjsmin==1.2.1
     # via django-compressor
-sqlparse==0.4.4
+sqlparse==0.5.0
     # via
     #   django
     #   django-debug-toolbar

requirements.txt

@@ -12,7 +12,7 @@ bleach==6.1.0
     # via -r requirements.in
 bleach-allowlist==1.0.3
     # via -r requirements.in
-certifi==2023.11.17
+certifi==2024.7.4
     # via requests
 cffi==1.16.0
     # via cryptography
@@ -27,7 +27,7 @@ cryptography==42.0.5
     #   josepy
     #   mozilla-django-oidc
     #   pyopenssl
-django==5.0.3
+django==5.0.8
     # via
     #   -r requirements.in
     #   django-registration
@@ -39,11 +39,11 @@ django-sass-processor==1.4
     # via -r requirements.in
 django-widget-tweaks==1.5.0
     # via -r requirements.in
-djangorestframework==3.14.0
+djangorestframework==3.15.2
     # via -r requirements.in
 huey==2.5.0
     # via -r requirements.in
-idna==3.6
+idna==3.7
     # via requests
 josepy==1.14.0
     # via mozilla-django-oidc
@@ -59,9 +59,7 @@ pyopenssl==24.1.0
     # via josepy
 python-dateutil==2.8.2
     # via -r requirements.in
-pytz==2023.3.post1
-    # via djangorestframework
-requests==2.31.0
+requests==2.32.0
     # via
     #   -r requirements.in
     #   mozilla-django-oidc
@@ -72,11 +70,11 @@ six==1.16.0
     #   python-dateutil
 soupsieve==2.5
     # via beautifulsoup4
-sqlparse==0.4.4
+sqlparse==0.5.0
     # via django
 supervisor==4.2.5
     # via -r requirements.in
-urllib3==2.1.0
+urllib3==2.2.2
     # via
     #   requests
     #   waybackpy