![]() The main problem is using too much heap memory, and the performance problem is likely to be due to excessive garbage collection when the remaining available heap is very small (but it's best to measure and profile to determine the exact cause of performance problems). I don't think that splitting this work onto multiple threads is going to provide much improvement, and may in fact make the problem worse by consuming even more memory. Do you know of a different approach to split at comas and replace the double quotes in each CSV line ? Would StringBuilder be of any healp here ? What about StringTokenizer ? How could I reduce the amount of heap memory used in the process ? Is the multithread implementation with Callable correct ? How could I improve the speed of the CSV reading ? Other than that the api is running out of heap memory when running on the server, I know that a solution would be to enhance the amount of available memory but I suspect that the replace() and split() operations on strings made in the Callable(s) are responsible for consuming a large amout of heap memory. To improve speed processing, I tried to implement multithreading with Callable(s) but I am not familiar with that kind of concept, so the implementation might be wrong. The code works fine on my local machine but it is very slow : it takes about 20 seconds to process 450 columns and 40 000 lines. I am not guaranteed to have the same header between files (each file can have a completly different header than another), so I have no way to create a dedicated class which would provide mapping with the CSV headers.Ĭurrently the api controller is calling a csv service which reads the CSV data using a BufferReader. It has to read big CSV files which will contain more than 500 columns and 2.5 millions lines each. I know we'll get to updating that importer at some point, it's just a matter of finding the best time to do it in the future.I am currently working on a spring based API which has to transform csv data and to expose them as json. Because there's only two of us working on it, we're picking features we feel we hear feedback on the most from users. Not sure when we'll get to that though, there's a lot of stuff we want to do to improve the site. Obviously less user friendly than a button that says Import from TappedOut, but a lot less tacky (also it'd work for every other deck list format). The most correct way to do this would be to allow users to make a layout, like a CSV or something, and import that way. We would like to improve our import options, the problem that we'd be trying to solve is updating the importer without explicitly having site specific importers. Sorry if we missed your suggestion, I try to keep an ear to the floor on most of the social medias, and our forums but sometimes stuff slips through. This tool/process does not work with alters since Archidekt does not support alters as far as I know. When you go to import the output you may get an error on archidekt, this is most commonly due to what set abbreviations the sites use so clear the set abbreviations from the cards in question and it should go through. ![]() ![]() Once the category is filled you can sort by category and copy to move all the blanks out of the way and copy just the output you need. Format B is harder as you have to paste the category into column B for each card.Format A you past into column a then use text to columns with a deliminator of # to fill the categories column.Import into the text editor on Archidekt.This step will depend on the formatting of the deck in tapped out.Second will either be blank or your desired Category and last will be this formula which creates the output to paste into Archidekt. First will be the input we copy from tapped out deck editor. ![]() In excel or gsheet (google's online excel) make 3 columns. So if your looking to migrate, this will help. Which made the transition harder than a simple copy/paste. One of my biggest hangups was I had a lot of custom categories due to how I build my decks. I spent yesterday exporting my decks from tappedout to Archidekt since Tapped out doesn't support edhrec. ![]()
0 Comments
Leave a Reply. |