php - How to find double transactions in MT940 -

at moment i'm working on import script imports bank account data mysql database using php. found mt940 pharser didn't wanted or didn't meet current mt940 standard. wrote own simple class parses me data need.

the problem no is, , might not mt940 issue, must filter double transaction. , in basic, simple, if exact same transaction exists in database, don't import again. that's have done.

but fun part: transactions might happen twice @ same day. example same transaction twice @ same day [someone might me , send me money twice ;)]. while importing first time, there no problem. in 1 file each transaction transaction.

but problem: because transaction not unique (mt940 doesn't send unique's transaction), hard filter out double transaction unique transaction. if have downloaded 2 mt940 files bank account. , 1 of 2 transactions in first file, , 1 in second file. when importing second file, tel me transaction double transaction.

so.. i'm struggling this, , i'm not one. big accountancy program's use mt940 structures , must deal same issue.

who knows way handle this?

additional information

in transaction (:61:), following information available:

• date
• amount
• bank account number
• bank account name
• bank account address
• transaction description

it might happen if transaction twice, information same.

example of 2 double transaction (of course modified name , bankaccount of transaction): (this 1 double in 1 mt940 file, 2 transactions).

:61:130311c000000000029,95n122nonref           0123456789 :86:/ordp//name/jansen w h/addr/someroad 1 9569 gs thecity/remi/n okia 3310/isdt/2013-03-11 

let's have 3 transactions: a, b , c.

a , b duplicates, therefore b must discarded

c different transaction, it's identical in every aspect.

by data provided in question, it's true = b = c. they're absolutely identical. there no ways determine, reading content, who's genuine , who's duplicate. of following scenarios may valid:

1. c duplicate of b, duplicate of a. discard b , c , keep a.
2. b genuine transaction identical a. c duplicate of b. discard b.
3. they're 3 genuine identical transactions. keep of them.
4. [etc...]

as can see, there's no algorithm can decide 100% case correct one. , since we're talking money, less 100% certainty can't accepted.

what then?

if computer can't determine it, let humans kick in. when record conflict, compile list. process not have conflicts , when you've finished batch, send conflict list human operator, , let him/her sort out duplicates genuine transactions (even if still wonder how can sort them if they're identical in every aspect)

if find out operators know how sort them, try deduce human algorithm, if possible.