先想清楚:什么叫"重复"First: What Counts as a "Duplicate"
去重前要先定义"重复"的标准——是整行一模一样才算重复,还是只要某几列相同就算?标准不同,结果天差地别。下面三种方法对应三种标准。
Before deduping, define what "duplicate" means — identical rows, or just matching key columns? The standard changes the result completely. The three methods below map to three standards.
方法一:按整行去重Method 1: Dedupe by Whole Row
最严格:只有当一行的所有列都相同时才判定为重复。适合"同一份数据被导入了两次"这类情况。
Strictest: a row is a duplicate only when every column matches. Ideal when the same dataset was imported twice.
方法二:按指定列去重(最常用)Method 2: Dedupe by Selected Columns (Most Common)
选一个或多个"关键列"(如订单号、手机号、邮箱),只要这些列相同就视为重复,保留其中一条。这是业务去重的主力方式。
Pick one or more "key columns" (order ID, phone, email); rows matching on those are duplicates, and one is kept. This is the workhorse of business dedup.
方法三:按"近似重复"去重Method 3: Dedupe Near-Duplicates
先对关键列做去空格、统一大小写、去掉符号等标准化,再判断重复。能抓出"张三"和" 张三 "这类肉眼难辨的重复。
First normalize the key columns — trim spaces, unify case, strip symbols — then compare. Catches sneaky duplicates like "张三" vs " 张三 ".
保留哪一条?首条还是末条Keep First or Last Occurrence?
| 选择Choice | 适用场景When to use |
|---|---|
| 保留首次出现Keep first | 后导入的是更新数据,想保留原始记录Later imports are updates; you want the original record |
| 保留末次出现Keep last | 后导入的是更新数据,想保留最新记录Later imports are updates; you want the newest record |
用 365Lab 去重(步骤)Dedupe with 365Lab (Steps)
- 在 解析页 导入 CSV。
- 打开 数据清洗 模块,勾选"去重"。
- 选择判断方式:全部列 / 指定列;并选择保留首条或末条。
- 必要时先用"去空格""大小写"等预处理,再用方法三。
- 预览确认后导出,去重操作可随时撤销。
- Import the CSV on the Parse page.
- Open the Clean module and tick "Deduplicate".
- Choose the basis: all columns or selected columns; keep first or last.
- If needed, pre-process with "trim" / "case" first, then use Method 3.
- Preview, export, and undo anytime.