Support Forum

[drew]

I am trying to TrackStudio for an existing project, I have dumped the bug/task information from the old tracking system and I am trying to populate the TrackStudio database (I'm using postgreSQL).

The first problem is:
I have been trying to do this by inserting into the gr_task table, with a little work, like fudging the task_id I can successfully insert into the gr_task table, unfortunately the new tasks do hot show up in TrackStudio. Obviously there has to be a different approach. Is there a stored procedure or something.

The second problem is:
I have added custom fields to the workflow I am using, how do I populate them?

Note I'm running TrackStudio 2.8.2.

Thanks in advance,

[admin]

I am trying to TrackStudio for an existing project, I have dumped the bug/task information from the old tracking system and I am trying to populate the TrackStudio database (I'm using postgreSQL).

By the way, you can do it with XML export/import (see sman, database management).

The first problem is:
I have been trying to do this by inserting into the gr_task table, with a little work, like fudging the task_id I can successfully insert into the gr_task table, unfortunately the new tasks do hot show up in TrackStudio. Obviously there has to be a different approach. Is there a stored procedure or something.

Some questions:
1) Do you have access rights (Access Control tab) to the parent of added tasks?
2) TrackStudio has extensive multi-level cache, you need restart server after manual database manipultaion.

Other approach - use internal API, but direct database/XML manipulation is simpler for this task.

The second problem is:
I have added custom fields to the workflow I am using, how do I populate them?

For each value you should create corresponding record in gr_udfsource (this table link custom field or custom field value to task, user, or workflow) and gr_udfval (value of custom field). In gr_udfval you should reference udfsource of related task.

The simplest way to learn - fill one or more values and export your database.

You can also contact me with IM if you have questions.

[drew]

I am trying to TrackStudio for an existing project, I have dumped the bug/task information from the old tracking system and I am trying to populate the TrackStudio database (I'm using postgreSQL).

By the way, you can do it with XML export/import (see sman, database management).

I tried creating XML for the import, however, the import failed trying to parse the file, the error is "Invalid byte 2 of 3-byte UTF-8 sequence". I thought I had any error somewhere in the xml so I stripped out all added udfsource and udfval recorts and all except about 500 tasks, this imported except that when I try to access the project I get an error "Task with identifier 'XXXX' not found. Possibly it has been removed." However, querying the gr_task table shows it as being there.

Either I have missed something, I am adding tasks to the gr_task table and entries to both the gr_udfsource and gr_udfval tables, in the second import attempt I simply added to the gr_task table.

One possibility is that the task_ids I am using are not valid, I'm using the old tracking systems bug id which is a long integer, which I convert to a string. How are the internal ids generated?

The first problem is:
I have been trying to do this by inserting into the gr_task table, with a little work, like fudging the task_id I can successfully insert into the gr_task table, unfortunately the new tasks do hot show up in TrackStudio. Obviously there has to be a different approach. Is there a stored procedure or something.

Some questions:
1) Do you have access rights (Access Control tab) to the parent of added tasks?
2) TrackStudio has extensive multi-level cache, you need restart server after manual database manipultaion.

Other approach - use internal API, but direct database/XML manipulation is simpler for this task.

The second problem is:
I have added custom fields to the workflow I am using, how do I populate them?

For each value you should create corresponding record in gr_udfsource (this table link custom field or custom field value to task, user, or workflow) and gr_udfval (value of custom field). In gr_udfval you should reference udfsource of related task.

The simplest way to learn - fill one or more values and export your database.

You can also contact me with IM if you have questions.

I have looked at the exported XML, I though I had everything worked out, with the exception of id's. The UTF-8 error confused me, but the task identifier error makes me think I am missing something.

Thanks in advance,

[admin]

I tried creating XML for the import, however, the import failed trying to parse the file, the error is "Invalid byte 2 of 3-byte UTF-8 sequence".

This is minor "feature" in sman, please try to change codepage in XML header to Windows-1252 (instead of UTF-8) manually. This is not problem with your data.

I thought I had any error somewhere in the xml so I stripped out all added udfsource and udfval recorts and all except about 500 tasks, this imported except that when I try to access the project I get an error "Task with identifier 'XXXX' not found. Possibly it has been removed." However, querying the gr_task table shows it as being there.

Please send me full stack trace from server console. Possible, you forget to add some object, that required for task (category, submitter, etc).

Either I have missed something, I am adding tasks to the gr_task table and entries to both the gr_udfsource and gr_udfval tables, in the second import attempt I simply added to the gr_task table.

One possibility is that the task_ids I am using are not valid, I'm using the old tracking systems bug id which is a long integer, which I convert to a string. How are the internal ids generated?

Plain simple id should be OK. We use GUID because it simplify data replication (we avoid situation when different records have some ID on different instances).

We use the following code to generate GUID:
id = (String) (new UUIDHexGenerator()).generate(null, null);

As far as I remember - UUIDHexGenerator is internal Hibernate class.

But I think that problem not in ID, we use simple id like '1' for root user and root task without problems.

[drew]

Sorry for the delay, we have been suffering network problems.

I tried creating XML for the import, however, the import failed trying to parse the file, the error is "Invalid byte 2 of 3-byte UTF-8 sequence".

This is minor "feature" in sman, please try to change codepage in XML header to Windows-1252 (instead of UTF-8) manually. This is not problem with your data.

Thank-you, thank-you, thank-you. I never would have figured this one out. It took me several attempts but I finally got the import to go through, took about an hour. However I still can not access the project.

I thought I had any error somewhere in the xml so I stripped out all added udfsource and udfval recorts and all except about 500 tasks, this imported except that when I try to access the project I get an error "Task with identifier 'XXXX' not found. Possibly it has been removed." However, querying the gr_task table shows it as being there.

Please send me full stack trace from server console. Possible, you forget to add some object, that required for task (category, submitter, etc).

I'm not sure what you mean by full stack trace. The Java Console shows two warnings about Style Sheets, nothing else.

The error has changed to just the word 'null'.

Note: I am using the WAR distribution, used the SA distribution to get sman for the import.

Either I have missed something, I am adding tasks to the gr_task table and entries to both the gr_udfsource and gr_udfval tables, in the second import attempt I simply added to the gr_task table.

One possibility is that the task_ids I am using are not valid, I'm using the old tracking systems bug id which is a long integer, which I convert to a string. How are the internal ids generated?

Plain simple id should be OK. We use GUID because it simplify data replication (we avoid situation when different records have some ID on different instances).

We use the following code to generate GUID:
id = (String) (new UUIDHexGenerator()).generate(null, null);

As far as I remember - UUIDHexGenerator is internal Hibernate class.

But I think that problem not in ID, we use simple id like '1' for root user and root task without problems.

The id's seem to import fine, and all the constraints activated so in that respect I think the import is fine. However, the error persists.

Any help would be appreciated.

[admin]

Thank-you, thank-you, thank-you. I never would have figured this one out. It took me several attempts but I finally got the import to go through, took about an hour. However I still can not access the project.

Yes, it can be long process - we recently import 75 MB file. By the way sman in 2.8.2 has major data import bugs, so I suggest you upgrade to 2.8.3 or simple download
http://www.trackstudio.com/tse-28/sman.jar

I'm not sure what you mean by full stack trace. The Java Console shows two warnings about Style Sheets, nothing else.

The error has changed to just the word 'null'.

Note: I am using the WAR distribution, used the SA distribution to get sman for the import.

Please try to start with sman and send me content of sman logging window when error occurs.

The id's seem to import fine, and all the constraints activated so in that respect I think the import is fine. However, the error persists.

Strange, but possbile. Please also try the following - left only several tasks in the XML and send it to me.

Yes, you must have task with id '1', user with id '1' and filter with id '1'.

[drew]

Thank-you, thank-you, thank-you. I never would have figured this one out. It took me several attempts but I finally got the import to go through, took about an hour. However I still can not access the project.

Yes, it can be long process - we recently import 75 MB file. By the way sman in 2.8.2 has major data import bugs, so I suggest you upgrade to 2.8.3 or simple download
http://www.trackstudio.com/tse-28/sman.jar

I'm not sure what you mean by full stack trace. The Java Console shows two warnings about Style Sheets, nothing else.

The error has changed to just the word 'null'.

Note: I am using the WAR distribution, used the SA distribution to get sman for the import.

Please try to start with sman and send me content of sman logging window when error occurs.

I started from within sman, it doesn't enable the 'Open window' button, but I just aimed my browser at http://127.0.0.1:8080/TrackStudio/index.html and connected successfully. I was also able to open the project in question. That said I am not sure whether the log will help much, but I will e-mailed that file to support@trackstudio.com right after this post (since the log is 9825 lines).

With the project loaded in the SA version, I noticed the memory footprint on my PC was about 756MB. This made me think that the Linux server that the WAR version is running on may not have sufficient memory, since it only has 256MB, I have memory on order it just hasn't arrived. I stripped another 256MB out of a different server, now I get the following error from Tomcat:

Code: Select all

HTTP Status 500 -

type Exception report

message

description The server encountered an internal error () that prevented it from fulfilling this request.

exception

javax.servlet.ServletException
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
   at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
   at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
   at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
   at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
   at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
   at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
   at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
   at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
   at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
   at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
   at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
   at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
   at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
   at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
   at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
   at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:432)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:386)
   at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:534)
   at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:530)
   at java.lang.Thread.run(Thread.java:536)

root cause

java.lang.OutOfMemoryError

Apache Tomcat/4.1.18

As I mentioned I have ordered more memory 1GB, but I am surprised as to the amount of memory required, this is one of the reasons we selected Linux over Windows as the OS. Is this an issue with Tomcat, should I consider a different application server?

The id's seem to import fine, and all the constraints activated so in that respect I think the import is fine. However, the error persists.

Strange, but possbile. Please also try the following - left only several tasks in the XML and send it to me.

Since the problem appears to be memory related, I am wondering how things will work as the task list grows, currently I have imported only about 8800 tasks moving forward there will be several users entering tasks live, the task list is likely to grow rapidly. Not to mention that since the old system would only export summary informatio, no messages or detail descriptions, the data I have used to ignite the new tracking system is relatively small, but going forward the tasks will have far more detail.

Yes, you must have task with id '1', user with id '1' and filter with id '1'.

All the id 1 items remain in the DB.

Thanks in advance,

[admin]

With the project loaded in the SA version, I noticed the memory footprint on my PC was about 756MB. This made me think that the Linux server that the WAR version is running on may not have sufficient memory, since it only has 256MB, I have memory on order it just hasn't arrived.
...
As I mentioned I have ordered more memory 1GB, but I am surprised as to the amount of memory required, this is one of the reasons we selected Linux over Windows as the OS. Is this an issue with Tomcat, should I consider a different application server?

The large amount of memory required at the server startup - because we reindex all tasks and messages for full text search. You can disable reindex at startup - just create empty file index\skipindex.flag in the upload directory.

Since the problem appears to be memory related, I am wondering how things will work as the task list grows, currently I have imported only about 8800 tasks moving forward there will be several users entering tasks live, the task list is likely to grow rapidly. Not to mention that since the old system would only export summary informatio, no messages or detail descriptions, the data I have used to ignite the new tracking system is relatively small, but going forward the tasks will have far more detail.

Yes, you need more memory. Currently we use 1GB RAM for ~10200 tasks (512MB is not enough). TrackStudio require large amount of memory because it use cache extensively for fast processing of hierarchical structures.

How many tasks do you have now and how many tasks your users enters every day ?

[admin]

As I mentioned I have ordered more memory 1GB, but I am surprised as to the amount of memory required, this is one of the reasons we selected Linux over Windows as the OS. Is this an issue with Tomcat, should I consider a different application server?

Important suggestions about java startup options - please use something like (we use jetty):
nohup /usr/local/jdk142/bin/java -server -Djava.awt.headless=true -classpath lib/jcert.jar:lib/jsse.jar:lib/jnet.jar:lib/javax.servlet.jar:lib/org.mortbay.jetty.jar:lib/xerces.jar:lib/jaxp.jar -XX:+UseConcMarkSweepGC -verbosegc -Xloggc:/tmp/gc.txt -XX:NewRatio=3 -Xmx768m -Xms512m -Dtrackstudio.Home=/home/maximkr/tsconfig -Djetty.home=. org.mortbay.jetty.Server

Important:
1) -Xmx should be less than available memory, some memory required for GC.
2) -XX:NewRatio=3 - avoid full gc when possible.
3) And use server JVM, of course.

[drew]

With the project loaded in the SA version, I noticed the memory footprint on my PC was about 756MB. This made me think that the Linux server that the WAR version is running on may not have sufficient memory, since it only has 256MB, I have memory on order it just hasn't arrived.
...
As I mentioned I have ordered more memory 1GB, but I am surprised as to the amount of memory required, this is one of the reasons we selected Linux over Windows as the OS. Is this an issue with Tomcat, should I consider a different application server?

The large amount of memory required at the server startup - because we reindex all tasks and messages for full text search. You can disable reindex at startup - just create empty file index\skipindex.flag in the upload directory.

If I understand correctly, we're on Linux, our upload directory is set to:

/usr2/TrackStudioData/uploads

There is already a skipindex.flag file in that directory, you are suggesting we creates another skipindex.flag in the directory:

/usr2/TrackStudioData/uploads/index

I tried this and still got the OutOfMemoryError.

Since the problem appears to be memory related, I am wondering how things will work as the task list grows, currently I have imported only about 8800 tasks moving forward there will be several users entering tasks live, the task list is likely to grow rapidly. Not to mention that since the old system would only export summary informatio, no messages or detail descriptions, the data I have used to ignite the new tracking system is relatively small, but going forward the tasks will have far more detail.

Yes, you need more memory. Currently we use 1GB RAM for ~10200 tasks (512MB is not enough). TrackStudio require large amount of memory because it use cache extensively for fast processing of hierarchical structures.

How many tasks do you have now and how many tasks your users enters every day ?

Obviously it will vary from day to day, I would expect this project to finish with less than 13000 task, as for the number of messages a minimum of 3 per task. So daily average would be less than 100 tasks and messages combined.

That said, do I need to consider even more than 1.2 GB of memory?

Thanks in advance,