This project I am working on requires the loading of almost 400,000 customers. We are doing these imports on small machines remotely so obviously speed is not a friend. I am running into a lot of problems and having run this procedure multiple times I have found tricks and tips that may help out to the world. Here is my list of what I have found

Emails are the key and have to be unique

Ok this is hopefully obvious to everyone, but in case its not here it is. We are coming from a system called Mainstreet and there are a lot of records with duplicate emails. Compiere will think all the records with the same email as the same business partner. My client has a test accounts with the same email address so when loaded into Compiere they all get loaded into a single business partner.

What I have done in my ETL tool of choice ( Talend ) is append the duplicate emails with “Duplicate: #” where # is the number incremented for each of the same email. I have delegating fixing the duplicate emails to the client.

Make sure there are some new records in your import pass or Compiere will run out of memory

I am using version Compiere 3.6.2, which has some bulk update changes. There is a line that checks for the size of the business partner map:
bpartnerMap.size()>COMMITCOUNT

The map only gets added with new business partners. I had a problem with our Oracle server and was constantly running out of space. Compiere just crapped out with no warning so I had to start the process over again. The process completed the main loop and was running the final updates and never got to the ad_user table inserts or updated the import table. I went to re-run the process and now the import was now running all updates and there was no inserts left so that piece of code above was never hit and so the other Maps were growing and growing until Compiere runs out of memory.

I modified the code and added a counter in the loop and if the loop % COMMITCOUNT == 0 then I run the saveBPartner code. I am running this now and I hope this works. It took a long time to track this one down.

Loading multiple addresses is possible, but make sure the value column is populated

I am running 2 passes of the import process to load the second address for business partner. The current system has a shipping and billing address so I load the shipping and then load the billing address. In my ETL tool, I was able to grab the C_BPARTNER_ID and put it in the import table hoping that would help out. Sadly it does not and the value column still needs to be populated. So if you are performing updates, mke sure the key ( value ) is populated.

Business Partners with no locations can cause problems with the imports

Some of the current systems data has business partners with no location data. Thats ok for this project, but Compiere’s import process may not like that. There is a part of code that sets the C_Location_ID to the MBPartnerLocation object and if its not set, an exception is thrown and not caught right away so basically the rest of the batch is thrown out. I added code in that section that basically continues the loop if no location is found so that no MBPartnerLocation object is created.

I am still loading business partners and continue to run into problems. Its a very frustrating experience and I can’t wait to pass this part of the project. If I find more tips, Ill update this entry. If you have any tips or tricks, let me know. I’d love to read them. I hate to think I am alone here.