Near Deduplication

NEAR DEDUPLICATION

Best for large-scale document reviews and/or messy data imports.

• Near dedupe differs from our standard “dedupe” as it goes beyond hash values (electronic thumbprints) in identifying duplicates, and/or similar documents
• Process can be done on native or imaged/produced data
• Steps of analysis:
  • Extract the text of every document
  • Text files are then analyzed through a “scoring” process
    • This creates one “Master” doc and scores all other nearly duplicative docs against it
    • Each score equates to a percentage of similarity
      • I.e. Document B is 97% similar to the Master-Document A
    • The client has full reign as to how stringent they want to be on what’s considered a Duplicate
      • I.e. Anything above 85% similar, could be considered a  duplicate in some cases, whereas another client may want to be more strict and set the threshold to 90% and above instead
• Once this scoring is complete, the results are viewable in Nextpoint
  • The Masters are isolated into one folder, and the Duplicates into another
  • The “Similarity Score” is brought into a coding field for visibility and sorting purposes
  • The Related Document window, will further show you a “cluster” of nearly duplicate documents
  • The Master will show up on the top level, with all other duplicates showing up as “related” documents underneath