🎓 Pragmatic AI Labs | Join 1M+ ML Engineers

🔥 Hot Course Offers:

• 🤖 Master GenAI Engineering - Build Production AI Systems
• 🦀 Learn Professional Rust - Industry-Grade Development
• 📊 AWS AI & Analytics - Scale Your ML in Cloud
• ⚡ Production GenAI on AWS - Deploy at Enterprise Scale
• 🛠️ Rust DevOps Mastery - Automate Everything

🚀 Level Up Your Career:

• 💼 Production ML Program - Complete MLOps & Cloud Mastery
• 🎯 Start Learning Now - Fast-Track Your ML Career
• 🏢 Trusted by Fortune 500 Teams

Learn end-to-end ML engineering from industry veterans at PAIML.COM

RDedupe

A Rust based deduplication tool

Goals

• Build a multiplatform, fast deduplication tool that uses Rust parallelization.

Current Status

• Added
• Added progress bar
• Added Polars DataFrame
• Added statistics about files with optional CSV report.

Future Improvements

• Add a GUI
• Add a web interface
• Fix GitHub Actions Build process to not fail silently!
• Store logs about actions performed across multiple runs

Building and Running

• Build: cd into rdedupe and run make all
• Run: cargo run -- dedupe --path tests --pattern .txt
• Run tests: make test

OS X Install

• Install rust via rustup
• Add to ~/.cargo/config

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

[target.aarch64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

• run make all in rdedupe directory