Fast_copy new release with new features

Major release. Introduces automatic filesystem detection, explicit hash algorithm selection, honest dedup accounting on link-incapable filesystems, improved duplicate-handling reporting, and several correctness fixes discovered during real-world testing against a Synology NAS. 289 automated tests (all passing) cover all 4 copy modes, FS detection, security hardening, resource leaks, MITM defenses, and pentest scenarios.

✨ New Features

Automatic filesystem detection

Detects the destination filesystem (ext4, btrfs, XFS, APFS, NTFS, FAT32, ReFS, HFS+, f2fs, zfs, NFS, SMB, and more) and probes its actual capabilities — hardlink, symlink, reflink CoW clones, case-sensitivity — before Phase 2. The detected strategy is now shown in the banner alongside the Dedup: line:

Dedup:       enabled (reflink)     ← on btrfs / XFS-reflink / APFS / ReFS
Dedup:       enabled (hardlink)    ← on ext4 / tmpfs / NTFS / etc.
Dedup:       enabled (none)        ← on FAT32 / exFAT (no links possible)

Uses cheap per-OS APIs (Linux /proc/self/mountinfo, macOS statfs(2), Windows GetVolumeInformationW) and runs targeted probes only for ambiguous filesystems (XFS reflink, NTFS Dev Drive, network mounts, FUSE). Adds ~5 ms per copy.

--hash=auto|xxh128|sha256 flag

Users can now explicitly choose the hash algorithm for dedup and verification:

• auto (default) — xxh128 if the xxhash package is installed, else sha256. Matches prior behavior.
• xxh128 — force xxh128 (10× faster, non-cryptographic). Errors with a clear install hint if xxhash is missing.
• sha256 — force cryptographic SHA-256 (collision-resistant). Recommended for adversarial environments.

The selected algorithm is printed in the main banner so the trust boundary is visible upfront:

Hash:        xxh128 (non-cryptographic; default)
Hash:        sha256 (cryptographic; forced)

-v / --verbose flag

Enables detailed FS detection output in the banner — FS type, capability matrix, detection and probe timings:

FS:          xfs → reflink
             hardlink=y symlink=y reflink=y case=sens
             detect=4.3ms probe=1.1ms (4 probes)

Honest dedup accounting on link-incapable filesystems

On FAT32/exFAT, dedup previously reported a misleading "Space saved: X MB" message even though duplicates were materialized as full copies. The new messaging is honest:

Dedup complete:
  Unique files:    44718
  Total duplicates: 46951 (51.2% of files)
  Bandwidth saved: 378.5 MB (transfer only)
  Disk usage:      888.2 MB (full copies — FS does not support links)

And the Phase 3 space check uses the full undeduplicated size to prevent unexpected ENOSPC errors mid-copy.

File-path destination for single-file copies

fast_copy host:file.tar.gz /local/renamed.tar.gz
fast_copy /data/photo.jpg /backup/photo-copy.jpg

Works across all four modes: L2L, R2L, L2R, R2R. A trailing / or \ forces directory interpretation. Detection uses splitext() so hidden directories (.config, .outputs) are not misinterpreted as file targets.

Improved duplicate-handling summary

Phase 6 now prints a clear per-type breakdown:

Duplicate handling:
  ✓ Hardlinks:          46951 (shared inode; zero extra disk)
  → all disk savings realized

Or on FAT32:

Duplicate handling:
  ✗ Full copies:            2 (FS does not support links — no disk savings)
  → no disk savings (bandwidth only)

🐛 Bug Fixes (HIGH severity)

Synology NAS: tar -T /dev/stdin permission denied

fast-copy's remote tar streaming used tar cf - --null -T /dev/stdin to pass the file list. On Synology DSM (and some other appliance OSes), paramiko's exec_command() can't open /dev/stdin via path resolution, causing all R2L and R2R operations against Synology to fail. Switched to tar -T - (read file list directly from stdin), which works universally across GNU tar, BSD tar, and busybox tar. Without this fix, fast-copy was completely broken against Synology NAS.

Stale manifest silent data loss

filter_unchanged_remote() trusted the .fast_copy_manifest.json as the source of truth for both file existence AND content hashes. If files were deleted at the destination between runs but the manifest remained (because rm * doesn't match dotfiles), fast-copy would report "DONE — All files up to date" and skip the entire copy while the destination was actually empty. Now always scans the remote for actual file existence; the manifest is used only as a hash cache.

🐛 Bug Fixes (MEDIUM severity)

• Windows verify crash on \\.\nul device paths
• FAT32 Phase 3 space check underreported required size, could trigger ENOSPC mid-copy

🔒 Security

fs_detect hardening

• _make_probe_dir() uses 128-bit entropy (was 32-bit), os.mkdir(mode=0o700) (not makedirs), and post-create lstat verification against symlink-swap races
• _cleanup_probe_dir() uses os.walk(followlinks=False) with per-entry lstat so injected symlinks are unlinked, never followed
• _walk_up_to_existing() rejects null-byte paths and tolerates symlink loops
• Linux ioctl(FICLONE) is architecture-guarded (skips probe on unrecognized arches)
• /proc/self/mountinfo parser decodes the documented octal escape sequences

MITM defenses verified

16 live attack tests confirm fast-copy's SSH host key verification:

• Wrong key planted in known_hosts → connection rejected
• --no-verify does NOT bypass host key checking
• TOFU prompt rejects on empty input / n / EOF
• No environment variable bypass (STRICT_HOST_KEY_CHECKING, FAST_COPY_INSECURE, etc.)
• Update download is HTTPS-only + pinned to GitHub domains + SSL CERT_REQUIRED enforced

Pentest scenarios

21 executable security scenarios covering symlink attacks, path traversal (filenames + tar members), race conditions, DoS (10K files, 100-deep nesting, circular symlinks, 1 GB sparse file, 100 hardlinks), DedupDB and manifest attacks, and fs_detect probe-directory attacks. All 21 pass; zero vulnerabilities found.

📋 Upgrade Notes

• No breaking behavioral changes for existing use cases. Existing command lines work identically.
• FAT32 / exFAT users: the dedup summary and space check now show the full (undeduplicated) size. This is a correctness fix; the previous numbers were misleading.
• Synology / DSM users: R2L / R2R / round-trip copies work for the first time.
• Stale .fast_copy_manifest.json: files will now actually get copied instead of a silent "up to date" report. This is a correctness fix.

📊 Test Coverage

289 automated tests across 8 test files, all passing:

| Suite | Tests |
|---|---:|
| test_v247.py (functional + FS detection integration) | 98 |
| test_all_args.py (every CLI flag) | 60 |
| test_fs_detect.py (detection + security) | 58 |
| pentest_scenarios.py (security scenarios) | 21 |
| test_mitm.py (live MITM defense) | 16 |
| test_dist_all_fs.py (per-filesystem) | 15 |
| test_fs_detect_leaks.py (resource leaks) | 11 |
| test_synology.py (live Synology DS720+ E2E) | 10 |