How I organize my digital files in 2024

Introduction

In this post, I describe how I keep my digital files organized, synchronized across platforms, and securely backed up.

This collection of file systems help me support very important life processes and goals:

• Be efficient at managing projects at work and at home.
• Manage archives of documents and media in the digital age.
• Manage relationships, communications, investments, civic life, self-development.

What’s new from the previous writeup

This is an update to previous posts I wrote in 2010 and in 2021–2022 on how I organize my digital documents.

Here are some of the recent major changes in my organization system:

• Why remote Git repositories are not backups, and why the content of Code/ should also be backed up like regular files.
• I added a home server to host files and services in my LAN but not on my workstation.
• Moved some archives to a new Deep archives/ directory.
• I started using the PARA method, especially in Documents/.
• I created an overview architecture diagram.

Use-cases and design goals

This section describes the context and constraints that guide my architecture design today.

• Necessity of working at home, remotely, and on business travels, across multiple devices.
• Necessity of having valuable files backed up to a Borg repository. Anything that is not backed up is considered a tmpfs.
• I value free and open-source software, for security and data sovereignty reasons.

Added benefits

In addition to supporting my work and my personal projects, my organization system bring additional side benefits.

• I can find anything fast.
• I can store new things fast.
• It is very hard to delete anything, even on purpose.
• I can do most organization tasks quickly using basic tools that work well for me:
  • The Nemo file manager on Linux Mint.
  • DigiKam for pictures management.

I rarely have to think about manually moving files around between systems. Nextcloud and Syncthing just automatically update files from where I collect and create them, to where I sort and organize them, to were I archive them. And it guarantees that any file I add to the system is appropriately backed up.

Overview

The diagram is shaped like a matrix, with horizontal and vertical lanes having these meanings:

• Platforms (verticals):
  • Vertical grey rounded rectangles represent the many redundant platforms and services supporting the directories.
• Directories (horizontals):
  • Each row represents a top-level directory of a conceptual hierarchy of files and subdirectories which is supported across several platforms.
  • These directories categorize the type of files they contain by the modality of access (how one uses the files, and witch application software one uses to access them), and the synchronization and backup requirements.

Platforms (verticals)

This section provides more detailed descriptions and explanations for the various redundant systems and platforms that support my file systems.

Nextcloud directories

The directories shown are the top-level directories in my Nextcloud account.

I usually edit and create files locally on my workstations, and let the Nextcloud desktop client synchronize the files to and from Nextcloud.

Workstation directories

I use conventional user directories (Documents/, Pictures/, etc.), plus a few of my own.

The Code/ directory, for example, is my own addition. I don’t like to have code repositories in my Documents/ PARA folders.

Phone directories

Personal files on my phone, such as camera pictures and browser downloads, are synchronized to my workstations using Syncthing. This way, I can edit and organize my phone files from my workstations.

For example, I like to organize photos on my desktop using DigiKam. As soon as I take a picture, it takes a few seconds for it to be downloaded to my computer. I can then sort them, make edits, export modified copies.

The Android system is prescriptive and rigid on the directory structure and the type of files they contain. The synchronization system that I use allows me to customize the filesystem-level target directories on the Android phone side and on the Linux workstation side.

The files are backed up to Borg through my workstation’s backup mechanism with Vorta (see section Vorta below).

The ‘home’ server.

This is a Debian jail on my TrueNAS server. It’s a “home base” in my LAN where I host personal files-related services.

Features:

• Mostly, I use SSHFS as the transport protocol to make any file operations.
• The underlying filesystem is a TrueNAS ZFS dataset.
• It is very suitable for hosting files in a way that I personally organize, with services and timers configured at the user level, with some isolation from the TrueNAS system.
• It allows me to have a faster-access copy of my Deep Archive/ directory. The Nextcloud server is in Montreal, QC, while I’m in Santa Fe, NM. The latency is quite noticeable, even over fast internet. Since many of my files are high-resolution video and photo media, accessing them over a 1 Gbps LAN in my own home network is much better.

There are three kinds of directories that I host there:

1. “Regular” directories.
   • Accessed via SSHFS.
   • I have a bookmark to sshfs://alex@home.alexware.deverteuil.net/home/alex in my Nemo file manager.
   • One important directory is ‘Deep archive’, which is quite massive (94GB), and is synchronized in Nextcloud.
   • Phone directories are also synchronized under ‘Sync’. This provides an additional Syncthing node for more efficient transfers, and acts as an “introducer” to streamline the setup of additional (or reinitialized) nodes.
2. Git repositories.
   • Instead of using private repositories in GitHub or GitLab, I just use Git over SSH in bare repositories on this host.
3. Borg repositories.
   • This is where all my backups go.
   • The borgbackup APT package is installed in the jail.
   • Borg uses SSH for network and transport security, and borg serve as the remote shell process.
   • I also need remote backups, so I have another ‘home’ host on my brother’s TrueNAS box the other side of the continent.

Directories (horizontals)

In this section, I expand and explain each top-level directory.

Code/

Code repositories have a very specific modality of access, which is through an IDE. They also don’t quite fit in the PARA folders. They often contain tens of thousands of small files that are sometimes rapidly modified (such as when checking out a Git branch), so I avoid synchronizing them in Nextcloud.

I do want to back them up to Borg though, to save any local un-merged changes. I don’t want to impose a tight discipline on myself for committing and pushing changes whenever I finish my work. I can just walk away from work when I’m done, knowing that my work-in-progress will be backed up.

Code repositories are categorized under 4 subdirectories:

• External/
  • Open-source projects.
  • Sometimes I just want to look at the source code and the code repository structure.
  • Occasionally, I’ll work on a PR.
  • Usually cloned from GitHub or some other public Git repository.
• Grafana/
  • Repositories for work.
  • Some public, some confidential.
  • Code for customer projects.
  • The remote “origin” is usually a public or private repository in Grafana Labs’ GitHub organization (https://github.com/grafana).
• Personal/
  • My own personal private repositories.
  • dotfiles, blog, scripts, systems administration playbooks and data.
  • The remote “origin” repository is usually home or a private Gitea server.
• Training/
  • tutorials
  • demos
  • boilerplates
  • I may or may not use those repos again, so I keep them separate.

Why Code should be backed up, and not rely on remote Git repositories:

• Dirty workspaces. There are always branches, modified files and stashes that are not published to the central Git repository.
• Demos and downloaded files. I don’t want to have to re-download everything and re-initialize my demos and lab environments, even if it is fully or partially automated.
• build files. Again, I’m probably using those, and I like to have those files available when I think of refering to those artifacts.
• Python venvs. They usually don’t get committed.
• Any other files excluded in .gitignore.
• When I migrate to a new workstation, I want things to pick up my work where I left in less than 10 minutes, not several days. First, I restore from a backup. Then I adjust the Git remotes.

Backups/

This is a repository of manually downloaded configuration and data exports for systems and apps that don’t have a fully automated backup system.
See also: How I organize my server backups in 2024.

For systems and applications that don’t have an automated backup, they usually have a way to export an archive containing configuration and user data via the GUI. All of those backup archives are organized here and are included in my Borg backups. They are also synchronized to Nextcloud, so that I can use them in systems administration tasks from any workstation.

Before and after I make any important configuration changes, I manually export a backup and save it here. I also do it just before migrating or re-installing a system.

Systems that are backed up this way:

• Appliances (pfSense, TrueNAS, Unifi controller).
• Applications and user data (Evolution, Joplin).
• Migrations to new workstations, or OS reinstalls (apt package lists, applications configs).

Content:

• Evolution (Gnome email, contacts and calendar client app).
• Flameshot config.
• TrueNAS configs.
• GnuCash configs and settings.
• Joplin backups (automated via a plugin).
• Linux Mint packages list.
• pfSense configuration.
• Tab Session Manager sessions.
• Vorta profile.

Documents/

My Documents directory organization structure is heavily based on the PARA method.
https://fortelabs.com/blog/para/

This set of PARA folders mirror my notebooks structure in my Joplin note-taking app. I use Joplin more intensively than my computer’s filesystem for Projects and Areas, but sometimes it makes more sense to store certain types of file on the computer’s filesystem.
See also: How I organize my notes app in 2024.

The directory structure looks like this:

• 1 Projets/
• 1 Projets art/
• 1 Projets écriture/
• 1 Projets Grafana Labs/
• 2 Areas/
  • Accounting/
  • Blog/
  • Car/
  • Home insurance/
  • Home office/ (printable template files, my CV, etc.)
  • Systems administration/
  • etc.
• 2 Grafana Labs/
• 2 Grafana Labs clients/
• 3 Références/
  • Mostly ebooks organized by topics of interest.
• 4 Archives/
  • Archive Amis/
  • Archive Art/
  • Archive Grafana Labs/
  • Archive Grafana Labs clients/
  • Archive Personnel/
• Maildir/

A few things to note:

• I split some of the PARA folders into contexts at the top-level (personal, art, work).
• Some directories are outside of the PARA model. These contain files managed by software.
  • The Maildir directory contains emails in Maildir format that I access via the Evolution mail client. Internally, it also follows the PARA structure, although it’s mostly archives that I host locally, to free up spaces in my remote mail accounts.

Inbox/

Several inboxes grouped together under one parent Inbox/ directory. In Digikam, I define it as a ‘Collection’, so that I can process and sort images incoming from all sources in the same place.

Symlinks to other directories:

• Cell phone camera/ → ~/Sync/Cell phone camera/
• Cell phone documents/ → ~/Sync/Cell phone documents/
• Cell phone downloads/ → ~/Sync/Cell phone downloads/
• Cell phone movies/ → ~/Sync/Cell phone movies/
• Cell phone pictures/ → ~/Sync/Cell phone pictures/
• Downloads/ → ~/Downloads/ – web browser downloads.

Other directories:

• OBS recordings
• Screenshots (Flameshot is configured to dump images here)
• Triage (temporary place for organizing photo albums)

Pictures/

This mostly contains organized archives and reference collections. Pictures files related to projects are handled in project directories in ‘Documents’.

Example directories here:

• Aquarelle/ (watercolor)
• Archive Grafana Labs/ (workplace social events)
• Archive Personnel/
  • Pictures of the last 4 years go here.
  • Older pictures go in Deep archive/.
• Archive Social/
• Collections/
• Dessin/
• Inspiration/
• Références/
  • Logos/ (collection of company and software logos to use in diagrams)
• Wallpapers/

Programs/

Executables I want to have, but that are not installed using a package manager. I keep my personal programs separate from externally downloaded programs. All those directories are added to my PATH environment variable.

Similar to the ‘Code’ directory, I separate ’external’ from ‘personal’ programs.

• external/
  • alloy (Grafana Alloy)
  • bw (Bitwarden CLI tool)
  • doctl (Digital Ocean CLI tool)
  • helm
  • hugo
  • k9s, kubectl and other Kubernetes tools
  • mimirtool
  • prometheus
  • promtool
  • yq
• personal/
  • various personal scripts written by me.

Templates/

This is a tiny directory. It only contains a handful of template files for my file manager’s right-click ‘Create New Document’ contextual menu action.

Sync/

This directory keeps all of my Syncthing target directories. Most of them are synchonized with directories on my Android phone. Most of those are processed like inbox directories, so I have symlinks to them in my ‘Inbox’ directory, so that I can access them in the proper context.

See also: Synchronizing files across phone, laptop and NAS with Nextcloud and Syncthing (2022).

Deep archive/

Over the years, my archives grew to a point where keeping those files synchronized was no longer desirable. Previously, I simply un-checked older Archive directories in the Nextcloud settings (known as “Remote-only archive directories” in my previous post). There were two drawbacks to this. 1) It made setting up Nextcloud sync on a new system slower and more error prone. 2) Those files were not included in my Borg backups.

My solution to this was to create a Deep archive/ directory in my Nextcloud account which would be synchronized on a server in my LAN instead of on my workstations.

This separate Deep archive/ storage achieves several benefits:

• Provides LAN access to large, rarely used files (faster throughput and lower latency compared to the Nextcloud server).
• Keeps Nextcloud synchronized folders settings more simple because I don’t have to keep excluding old archives.
• Files are now included in Borg backups.

Here is what my Deep archive contains:

• Archive Arts Plastiques documents/
• Archive Arts Plastiques photos/
• Archive BLC documents/
• Archive BLC photos/
• Archive CQFA documents/
• Archive CQFA photos/
• Archive IKEA photos/
• Archive Informatique Appliquée documents/
• Archive iWeb documents/
• Archive Jetsgo documents/
• Archive Jetsgo photos/
• Archive Parkour photos/
• Archive Personnel photos/
  • Photos older than the ~4 last years go here.
• Archive Toastmasters documents/

As you can see, it contains photos and documents archives from my “past lives”.

I use nextcloudcmd to synchronize it with Nextcloud, and Borgmatic to backup files to the local and remote borg servers.

I initialized my local copy of the files with an Rsync from Nextcloud’s data storage. This was the fastest and most reliable network copy method for the initialization. The first nextcloudcmd run took 6 hours. even though there were no differences, just to create the initial files metadata index. Subsequent runs take about 5 minutes. I set the systemd timer to 10 minutes.

Videos/

My Videos/ directory is similar in the way I access files to Deep archive/. It contains large files that I access rarely. I would rather access them via my LAN than over the Internet. I sometimes want them on my workstation directly, but only when actively working and editing files. Streaming Videos using Filebrowser or SSHFS works really well over the LAN.

Software and platform components

In this section, I provide details of the various software and platform components that I use, as well as factors that helped me decide to use them.

Nextcloud

https://nextcloud.com/
https://github.com/nextcloud/all-in-one

Nextcloud is my main personal data platform. I use it for all my files, as well as my calendar and contacts.

Features:

• The desktop client for Linux is very good at two-way synchronization of multiple directories.
• The Android client and the web UI allow me to access any files from anywhere.
• I use the file sharing features a lot with friends, family and colleagues.

See also: Nextcloud self-hosted architecture.

Vorta

https://vorta.borgbase.com/
https://www.borgbackup.org/

Vorta is a GUI program for Borgbackup. It’s ideal for backing up my personal user files to a self-hosted Borg repository.

Here’s the list of exclusion rules in my Vorta for Borgbackup config.

/home/alex/Downloads/
/home/alex/Public/
/home/alex/snap/
/home/alex/mnt/
/home/alex/tmp/
/home/alex/.cache/
/home/alex/.config/*/Cache/
/home/alex/.var/*/Cache/
/home/alex/.var/*/cache/
/home/alex/.googleearth/Cache/
/home/alex/.config/Slack/Service Worker/CacheStorage/
/home/alex/.local/share/Trash/
/home/alex/.local/share/flatpak/

Everything else in /home/alex/ is backed up. This includes:

• All the dotfiles.
• Code/ repositories.
• Backups/, Documents/, Inbox/, Pictures/, Programs/, Sync/.

Borgmatic

https://torsion.org/borgmatic/
https://www.borgbackup.org/

Another client software for Borgbackup. This one operates with configuration files and a command-line interface. I use it to backup servers, system files, and personal files hosted on the headless ‘Home’ server.

Syncthing

https://syncthing.net/
https://github.com/Catfriend1/syncthing-android

I use Syncthing desktop, Syncthing server, and Syncthing-Fork for Android.

See also Synchronizing files across phone, laptop and NAS with Nextcloud and Syncthing.

DigiKam

https://www.digikam.org/

This is my main tool for organizing and sorting pictures and images.

I define three directories as ‘Collections’:

• Documents/
  • For projects that involve working with image files, I want the option to use Digikam.
• Inbox/
  • Capturing and sorting pictures from my cell phone camera.
• Pictures/
  • Organizing and archiving pictures in collections.

I configure DigiKam to update the pictures metadata, so that there is no dependency on DigiKam’s database, and all updates are synchronized to other workstations via files synchronization.

Nemo

https://en.wikipedia.org/wiki/Nemo_(file_manager)

Nemo is the file manager included with Linux Mint.

It has a pretty slick Nextcloud integration. It adds a small overlay icons on top of every synchronized file and directory indicating the synchronization status. It also allows me to right-click on a file or a directory to access the sharing features.

For quick access to Deep archive/ and Videos/ hosted on my ‘home’ server, I just need to create a bookmark for ssh://alex@home.<network>.net/home/alex in Nemo.

See also

• How I organize my server backups in 2024.
• How I organize my notes app in 2024.
• How I organize my home directory (2021–2022).
• How I organize my digital documents (2010–2020).
• Synchronizing files across phone, laptop and NAS with Nextcloud and Syncthing (2022).
• Journaling TOC (index of 2019–2021 posts related to paper-based organization).
• Firefox tabs organized (2021).