GitHub - SpeedcubeDE/speedcube.de-forum-archive: public information on the archived Speedcube.de-Forum (forum.speedcube.de/speedcubers.de) · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
SpeedcubeDE
speedcube.de-forum-archive
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>7 Commits<br>7 Commits
README.md
README.md
View all files
Repository files navigation
The Speedcube.de forum used to be a German Speedcubing forum based on the MyBB forum software.
It was archived in September of 2024 and replaced by a static read-only version.
Archiving was done by running the following wget command against the forum:
wget \<br>--recursive \<br>--level inf \<br>--page-requisites \<br>--convert-links \<br>--adjust-extension \<br>--restrict-file-names=windows \<br>--wait 0 \<br>--span-hosts --domains=forum.speedcube.de,speedcubers.de \<br>--trust-server-names \<br>https://forum.speedcube.de/
The above command was chosen based on these considerations:
Don't use --mirror and instead specify --resursive and --level inf separately, because --mirror also implies --timestamping,<br>and timestamping was not supported by the server anyway.
Use --page-requisites, --convert-links and --adjust-extension to make the archive completely self-contained and statically servable.
Use --restrict-file-names=windows to make the converted filenames more portable. Notably to replace the query parameters' ? with @.
Use -wait 0 to speed up the process, since the old server did not impose any rate limits anyway.
Use --span-hosts --domains=forum.speedcube.de,speedcubers.de to also traverse speedcubers.de, which just redirects back to forum.speedcube.de.
Use --trust-server-names to have the speedcubers.de redirects render as their redirect target in the produced archive,<br>so the redundant redirect hops through speedcubers.de become regular relative links in the final result.
The --no-clobber option would have been used to be able to resume partial downloads by having wget respect and use files already existing on disk.<br>However, this does not work as wget claims the option is incompatible with --convert-links.<br>I was unable to get wget to resume any partial downloads and had to start over each time my hard drive got full.<br>See also https://savannah.gnu.org/bugs/index.php?62782#comment2 for a comment of mine on the gnu issue tracker.
The command took 1 week to traverse the entire forum and download all files, and 1 more week to convert all links.<br>The majority of the slowdown was caused by wget having to parse posts in threaded mode,<br>which are not only many in number, but also each very large. See for example Thread #7728.
Because the produced mirror is around 260GB large, and buying a server with that much disk space just for serving it would be expensive,<br>the final mirror was compressed into a SquashFS file instead:
mksquashfs forum.speedcube.de forum_speedcube_de_archive.sqsh -comp zstd
The file can be downloaded at forum.speedcube.de/forum_speedcube_de_archive.sqsh.
SquashFS not only supports good compression (I used zstd in this case) but also efficient random access.<br>The resulting file only measures ~12GB can then be mounted, allowing access to the individual files again.<br>This requires ad-hoc decompression of each accessed file, which is well worth it.
mkdir /mnt/forum_speedcube_de_archive<br>mount -t squashfs -o loop forum_speedcube_de_archive.sqsh /mnt/forum_speedcube_de_archive
This line was added to /etc/fstab to automatically mount the archive at boot:
/path/to/forumbackup/forum_speedcube_de_archive.sqsh /mnt/forum_speedcube_de_archive squashfs defaults 0 0
The produced static files are now served via a nginx config like this:
server {<br>server_name forum.speedcube.de;
# rewrite does not work with query params<br># so we're using ifs and returns instead.
# Some Actions (e.g. action=lastpost) used to dynamically determine where to navigate,<br># and then navigate to an actual canonical...