Support Forum

[admin]

If I understand correctly, we're on Linux, our upload directory is set to:

/usr2/TrackStudioData/uploads

There is already a skipindex.flag file in that directory, you are suggesting we creates another skipindex.flag in the directory:

/usr2/TrackStudioData/uploads/index

I tried this and still got the OutOfMemoryError.

What amount of memory you have now ? What JDK you use ?
Please also send me your server startup options.

Obviously it will vary from day to day, I would expect this project to finish with less than 13000 task, as for the number of messages a minimum of 3 per task. So daily average would be less than 100 tasks and messages combined.

That said, do I need to consider even more than 1.2 GB of memory?

No, 1 GB should be enough for 15000-20000 tasks. We was run ~8000 tasks and 25000 messages on 512 MB without major problems, but for 10000 1GB is better, just for decrease GC time.

Please also try to tune WEB-INF/classes/cache.ccf to decrease
jcs.default.cacheattributes.MaxObjects=10000
and
jcs.default.elementattributes.MaxLifeSeconds=7200.

Actually, TrackStudio need not too much memory to run. But it stores objects in cache during MaxLifeSeconds, after GC can collect this objects. Problems occurs when TrackStudio store too many objects and GC can't collect them because their life time below MaxLifeSeconds. After full GC memory requirements drops to 150-200 MB for our database.

The another point - many projects with deep hierarchy require less memory for caching, then 1 project with many tasks.

PS. How you populate long (2000+ chars) task description/message description ? You should use GR_LONGTEXT for them.

[drew]

If I understand correctly, we're on Linux, our upload directory is set to:

/usr2/TrackStudioData/uploads

There is already a skipindex.flag file in that directory, you are suggesting we creates another skipindex.flag in the directory:

/usr2/TrackStudioData/uploads/index

I tried this and still got the OutOfMemoryError.

What amount of memory you have now ? What JDK you use ?
Please also send me your server startup options.

The machine has a 2.8GB P4 with 512MB RAM running SuSE Linux 8.2 with 2.0GB swap.

Runing Sun's JRE 1.4.1 as follows:

Code: Select all

/usr/lib/SunJava2-1.4/bin/java -XX:+UseConcMarkSweepGC -verbosegc -Xloggc:/tmp/gc.txt -XX:NewRatio=3 -Xmx384m -Xms256m -Djava.endorsed.dirs=/opt/jakarta/tomcat/bin:/opt/jakarta/tomcat/common/endorsed -classpath /usr/lib/SunJava2-1.4/lib/tools.jar:/opt/jakarta/tomcat/bin/bootstrap.jar -Dcatalina.base=/opt/jakarta/tomcat -Dcatalina.home=/opt/jakarta/tomcat -Djava.io.tmpdir=/opt/jakarta/tomcat/temp org.apache.catalina.startup.Bootstrap start

With the above command line trying to open the project seem to hang for about 20 minutes and then gave the 'null' error. Below is a snapshot of top while the open request was running:

Code: Select all

top - 19:26:55 up  5:42,  2 users,  load average: 1.38, 0.92, 0.60
Tasks: 155 total,   2 running, 153 sleeping,   0 stopped,   0 zombie
Cpu(s):   5.3% user,   0.6% system,   0.0% nice,  94.2% idle
Mem:    513808k total,   496176k used,    17632k free,    11656k buffers
Swap:  2104504k total,      432k used,  2104072k free,   103948k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command                                                           
11181 wwwrun    25   0  351m 351m  14m R 63.5 70.0   4:07.45 java                                                               
11182 wwwrun    15   0  351m 351m  14m S 27.9 70.0   1:27.32 java                                                               
11291 root      19   0   852  852  612 R 27.9  0.2   0:00.83 top                                                               
11216 wwwrun    15   0  351m 351m  14m S  7.7 70.0   0:50.72 java                                                               
    1 root      15   0   104  104   68 S  0.0  0.0   0:03.76 init                                                               
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration_CPU0                                                     
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration_CPU1                                                     
    4 root      15   0     0    0    0 S  0.0  0.0   0:00.00 keventd                                                           
    5 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd_CPU0                                                     
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd_CPU1                                                     
    7 root      15   0     0    0    0 S  0.0  0.0   0:00.14 kswapd                                                             
    8 root      25   0     0    0    0 S  0.0  0.0   0:00.00 bdflush                                                           
    9 root      15   0     0    0    0 S  0.0  0.0   0:00.07 kupdated                                                           
   10 root      15   0     0    0    0 S  0.0  0.0   0:00.00 kinoded                                                           
   12 root      25   0     0    0    0 S  0.0  0.0   0:00.00 mdrecoveryd                                                       
   16 root      15   0     0    0    0 S  0.0  0.0   0:00.06 kreiserfsd                                                         
   54 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 lvm-mpd                                                           
  413 root      15   0   572  572  476 S  0.0  0.1   0:00.49 syslogd                                                           
  416 root      15   0  1296 1296  364 S  0.0  0.3   0:00.09 klogd                                                             
  461 root      21   0     0    0    0 S  0.0  0.0   0:00.00 khubd                                                             
  636 root      25   0   276  264  216 S  0.0  0.1   0:00.01 resmgrd                                                           
1000 bin       25   0   288  280  220 S  0.0  0.1   0:00.00 portmap                                                           
1030 at        15   0   328  308  260 S  0.0  0.1   0:00.00 atd                                                               
1084 postgres  15   0  1924 1924 1840 S  0.0  0.4   0:00.03 postmaster                                                         
1088 postgres  15   0   732  472  416 S  0.0  0.1   0:00.00 postmaster                                                         
1089 postgres  15   0   972  728  616 S  0.0  0.1   0:00.01 postmaster

Obviously it will vary from day to day, I would expect this project to finish with less than 13000 task, as for the number of messages a minimum of 3 per task. So daily average would be less than 100 tasks and messages combined.

That said, do I need to consider even more than 1.2 GB of memory?

No, 1 GB should be enough for 15000-20000 tasks. We was run ~8000 tasks and 25000 messages on 512 MB without major problems, but for 10000 1GB is better, just for decrease GC time.

512GB does not seem to be sufficient in this case, but this is a flat (hierarchy) project.

Please also try to tune WEB-INF/classes/cache.ccf to decrease
jcs.default.cacheattributes.MaxObjects=10000
and
jcs.default.elementattributes.MaxLifeSeconds=7200.

I decreased these values to the following:

Code: Select all

jcs.default.cacheattributes.MaxObjects=5000
jcs.default.elementattributes.MaxLifeSeconds=1800

with the same results as above, ~20 minutes ant then the 'null' error.

Actually, TrackStudio need not too much memory to run. But it stores objects in cache during MaxLifeSeconds, after GC can collect this objects. Problems occurs when TrackStudio store too many objects and GC can't collect them because their life time below MaxLifeSeconds. After full GC memory requirements drops to 150-200 MB for our database.

The another point - many projects with deep hierarchy require less memory for caching, then 1 project with many tasks.

The hierarchy is flat by it's very nature, bugs issues against a single project, I can possibly try to restructure the project to have a secondary levels, but not sure I could go beyond that. Certainly the current error and load time connecting with the SA version (remotely) is enough to encourage building the hierarchy.

PS. How you populate long (2000+ chars) task description/message description ? You should use GR_LONGTEXT for them.

I did not import any long text as the old system would not let me export more than summary information.

Thanks in advance,

[admin]

[quote="drew"]
The machine has a 2.8GB P4 with 512MB RAM running SuSE Linux 8.2 with 2.0GB swap.

Runing Sun's JRE 1.4.1 as follows:

Code: Select all

/usr/lib/SunJava2-1.4/bin/java -XX:+UseConcMarkSweepGC -verbosegc -Xloggc:/tmp/gc.txt -XX:NewRatio=3 -Xmx384m -Xms256m


Please
1) Increase -Xmx384m to -Xmx512m
2) Wait until (if) it crash and send me content of /tmp/gc.txt. It shows how Java use memory dynamically.

If you have single list of bugs in one project - this means that TrackStudio should load (and store) all tasks into the memory to filter/sort them. If you have deep hierarchy - TrackStudio can load only tasks for current active project and free memory after some inactivity time.
I'll check how it performs with one project and many tasks today.

[admin]

I has been investigated this problem, the core of it is too plain hierarchy.

I wrote following simple program (you need gcc to compile it):

Code: Select all

#include <stdio.h>
#include <stdlib.h>

int random(float max) {
        return 1+(int) (max*rand()/(RAND_MAX+1.0));
}

void main()
{
        for (int i=3;i<15000;i++) {
        printf("<row>\n");
        printf(" <data name=\"task_id\"><![CDATA[%d]]></data>\n",i);
        printf(" <data name=\"task_shortname\">NULL</data>\n");
        printf(" <data name=\"task_name\"><![CDATA[TestBug N%d]]></data>\n",i);
        printf(" <data name=\"task_submitdate\" type=\"timestamp\"><![CDATA[2004-01-21 13:56:42]]></data>\n");
        printf(" <data name=\"task_updatedate\" type=\"timestamp\"><![CDATA[2004-01-21 13:56:42]]></data>\n");
        printf(" <data name=\"task_closedate\">NULL</data>\n");
        printf(" <data name=\"task_description\">NULL</data>\n");
        printf(" <data name=\"task_abudget\">NULL</data>\n");
        printf(" <data name=\"task_budget\">NULL</data>\n");
        printf(" <data name=\"task_deadline\">NULL</data>\n");
        printf(" <data name=\"task_category\"><![CDATA[%d]]></data>\n",random(5));
        printf(" <data name=\"task_status\"><![CDATA[%d]]></data>\n",random(5));
        printf(" <data name=\"task_resolution\">NULL</data>\n");
        printf(" <data name=\"task_priority\"><![CDATA[%d]]></data>\n",random(5));
        printf(" <data name=\"task_submitter\"><![CDATA[2]]></data>\n");
        printf(" <data name=\"task_handler\"><![CDATA[2]]></data>\n");
//        printf(" <data name=\"task_parent\"><![CDATA[1]]></data>\n");
        printf(" <data name=\"task_parent\"><![CDATA[%d]]></data>\n",random(i-1));
        printf(" <data name=\"task_longtext\">NULL</data>\n");
        printf(" <data name=\"task_number\"><![CDATA[%d]]></data>\n",i);
        printf("</row>\n");
        }

}


This program generate XML for 15000 tasks, where parent for each next task is randomly selected one of the existing tasks. You should add generated tasks to the exisitng 2 tasks in the export XML.

I test TrackStudio 2.8.3 on such database on standard (poor for production use) configuration:
1) HSQL as database
2) JDK 1.3.1, hotspot JVM (not server)
3) Mx limited to 384MB, no other optimization options.
4) Pentium-4, 2.5GHz, 512MB RAM
5) Windows XP.

And its run fine for me - about 20-30 seconds for start and generally less then second during the work.

You can download pre-generated export XML file here (320KB) for testing purpose:
http://www.trackstudio.com/tse-28/exp.zip

But if you change this program and set root task as parent for all task - it become very slow, throw OutOfMemory exception, etc. This is because TrackStudio need to store all those tasks to filter and sort them, generate reports, etc.

From my expirence you should avoid more then 500 tasks per project, I think that 1000 tasks (direct children) per project is upper limit. I suggest you divide your data into several subprojects by some meaningful criteria - by submit date, by submitter, by status on data transfer date, etc.

Hope, this helps.

[drew]

The machine has a 2.8GB P4 with 512MB RAM running SuSE Linux 8.2 with 2.0GB swap.

Runing Sun's JRE 1.4.1 as follows:

Code: Select all

/usr/lib/SunJava2-1.4/bin/java -XX:+UseConcMarkSweepGC -verbosegc -Xloggc:/tmp/gc.txt -XX:NewRatio=3 -Xmx384m -Xms256m


Please
1) Increase -Xmx384m to -Xmx512m
2) Wait until (if) it crash and send me content of /tmp/gc.txt. It shows how Java use memory dynamically.

I have tried these settings with both the original cache.ccf settings and the modified values, in both cases after more than an hour the open project request does not return. In between each test I stopped and restarted Tomcat.

If you have single list of bugs in one project - this means that TrackStudio should load (and store) all tasks into the memory to filter/sort them. If you have deep hierarchy - TrackStudio can load only tasks for current active project and free memory after some inactivity time.
I'll check how it performs with one project and many tasks today.

Currently I am trying to restructure the XML so that the tasks will are stored in a hierarchy within the project in question. Due to the size of the import file, this will take some time. My concern, presuming more memory resolves the open issue, is that with the flat hierarchy the performance will not be acceptable.

As a side project I was thinking of switching to JRockit since it is reported to have better performance than Tomcat (please no flames), hopefully this does not increase the memory requirements. Does this seem reasonable/advisable?

As with all things you can't wait for, the new memory seems to be lost in transit.

Again thanks in advance,

[admin]

I have tried these settings with both the original cache.ccf settings and the modified values, in both cases after more than an hour the open project request does not return. In between each test I stopped and restarted Tomcat.

Yes, I have the same results during today testings - 15000 tasks per project is too much for TrackStudio, optimization options can't help in such case.
And I have no ideas why - we use plain simple Java classes (enchanced by Hibernate), no EJB, etc. Its difficult for me to understand how 15000 tasks can eat 1GB of RAM, I'll try to investigate this.

Currently I am trying to restructure the XML so that the tasks will are stored in a hierarchy within the project in question. Due to the size of the import file, this will take some time. My concern, presuming more memory resolves the open issue, is that with the flat hierarchy the performance will not be acceptable.

I think you should restructurize it in some simple way. For example, create subproject for each submitter, remember ID of those subprojects and assign task_parent to the corresponding project when you generate XML. Also I suggest you test your server with .zip mentioned in previous message, hope this helps.

As a side project I was thinking of switching to JRockit since it is reported to have better performance than Tomcat (please no flames), hopefully this does not increase the memory requirements. Does this seem reasonable/advisable?

From my expirence JRockit (instead of Sun JDK, I think, not Tomcat ) can gain additional 20% in perfomance, but it is not enough in your case. In our case (RH9, Jetty) JRockit was unstable and crash daily (because some Jetty-JRockit problems). Possible, it will perform better with Suse&Tomcat.

In the next version we'll implement total query caching, it can gain 2x-3x perfomance burst in many cases.

[drew]

Currently I am trying to restructure the XML so that the tasks will are stored in a hierarchy within the project in question. Due to the size of the import file, this will take some time. My concern, presuming more memory resolves the open issue, is that with the flat hierarchy the performance will not be acceptable.

I think you should restructurize it in some simple way. For example, create subproject for each submitter, remember ID of those subprojects and assign task_parent to the corresponding project when you generate XML. Also I suggest you test your server with .zip mentioned in previous message, hope this helps.

Well the good new is that the 1GB of memory showed up, so now the machine has 1.2GB RAM.

While I was waiting for the memory I restructured the XML to build a shallow hierarchy, there are a few trees with more than 500 tasks but all seem to respond well. The import was clean, i.e. there was no failures. We could login, decend a large portion of the tree.

The bad news is that we had one task, out of 8809 the had a null submitter, I'm surprised there is no constraint to prevent this at the import level since it is obviously a requirement. It would also have saved me a lot of time.

Basically when we tried to open that branch (subproject) containing the bad task we got a "General TrackStudio Error", not very informative. What we discovered was that we could jump to individual tasks via the alias and the jump widget. So it became the tedious task of jumping to each task in the branch until one produced a "null" error. When back to the import file and analyzed that task. Once we noticed the task_submitter was NULL we updated the field via psql, restarted Tomcat and all was well.

I have to admit that I was pleasantly surprised at how well the performance is, I was beginning to have my doubt after that last few days! I was even beginning to wonder if 1.2GB was going to be sufficient. But, login is about 15 seconds and walking the branches after that is instantaneous.

As a side project I was thinking of switching to JRockit since it is reported to have better performance than Tomcat (please no flames), hopefully this does not increase the memory requirements. Does this seem reasonable/advisable?

From my expirence JRockit (instead of Sun JDK, I think, not Tomcat ) can gain additional 20% in perfomance, but it is not enough in your case. In our case (RH9, Jetty) JRockit was unstable and crash daily (because some Jetty-JRockit problems). Possible, it will perform better with Suse&Tomcat.

It turns out that JRockit does not support the glibc (version 2.3.2) that ships as part of SuSE 8.2. So I have stayed with Tomcat, I certainly can not complain the system is not humming!

In the next version we'll implement total query caching, it can gain 2x-3x perfomance burst in many cases.

Sounds like the next version should smoke!

Thanks for all your help, it is a pleasure to deal with support the obviously has a passion for their product, not to mention a excellent understanding of the workings of that product. Hats off to you!

[admin]

Well the good new is that the 1GB of memory showed up, so now the machine has 1.2GB RAM.

While I was waiting for the memory I restructured the XML to build a shallow hierarchy, there are a few trees with more than 500 tasks but all seem to respond well. The import was clean, i.e. there was no failures. We could login, decend a large portion of the tree.

The bad news is that we had one task, out of 8809 the had a null submitter, I'm surprised there is no constraint to prevent this at the import level since it is obviously a requirement. It would also have saved me a lot of time.

Basically when we tried to open that branch (subproject) containing the bad task we got a "General TrackStudio Error", not very informative. What we discovered was that we could jump to individual tasks via the alias and the jump widget. So it became the tedious task of jumping to each task in the branch until one produced a "null" error. When back to the import file and analyzed that task. Once we noticed the task_submitter was NULL we updated the field via psql, restarted Tomcat and all was well.

Thank you for the report, we'll check all fields for NULL/NOT NULL in the next version.

I have to admit that I was pleasantly surprised at how well the performance is, I was beginning to have my doubt after that last few days! I was even beginning to wonder if 1.2GB was going to be sufficient. But, login is about 15 seconds and walking the branches after that is instantaneous.

15 seconds for login is not very fast Is it only first login takes 15 seconds or every login ?

Thanks for all your help, it is a pleasure to deal with support the obviously has a passion for their product, not to mention a excellent understanding of the workings of that product. Hats off to you!

Thank you Can I add your comment to the our "Testimonials" page ?

[drew]

15 seconds for login is not very fast Is it only first login takes 15 seconds or every login ?

First login, second time is about 4 seconds , please note this is a remote login, not local so there is WAN latency.

Thanks for all your help, it is a pleasure to deal with support the obviously has a passion for their product, not to mention a excellent understanding of the workings of that product. Hats off to you!

Thank you Can I add your comment to the our "Testimonials" page ?

Yes, you may.