I guess my suggestion does not increase the speed tremendously. I just tried by hand. I might notice a small increase in query speed. Therefore, it must be other reason (like "Bug"). And, I do see your point about getting all information (in the way this code (BrFileCatalogInfo) was created/coded. ). Hiro Kris Hagel wrote: > Hello, > I guess I qualify as the maintainer of the file catalog code. > I changed your point # 1 and I removed the print statements. They > were put there for original debugging and never removed. It should > clean things up. > > I did not for the moment change the Select *. There are two reasons. > One is that the GetMultiple method of BrRdbmDb does not lend itself to > doing that. This could be changed, but requires time for testing > because GetMultiple is used EVERYWHERE in the DB infrastructure and > one needs to be very careful not make second order effects. I would > rather spend that time on data analysis. Perhaps more importantly, > the idea is to get all of the data from the db into the BrFileInfo > structure so that it can be used if necessary. That way, the user > will not have to decide in advance what part of the data in the file > catalog for a given run he wishes to use. Once he gets the data for a > given run, he has it at his disposal. Furthermore, I cannot imagine > that getting the script, creator and comments is what is bogging us > down. It is not that much data on the scale of what our network can > handle or what I think mysql should be able to handle. I see the > problem more likely in infinite loops with connections not being > closed or something like that. The above changes might fix it, but I > am not sure. > > I have made the changes in my brat. I am going to commit my brat with > many changes in the next few days after doing a few more tests. > > Kris > > Truls Martin Larsen wrote: > >> Hi, >> >> I realize there was a "bug" in the code... It had an exponential growth >> in the number of queries... doing regsister for a single run, the >> general update() in a loop... >> >> Sorry for slowing thing down... it could have been avoided if the kInit >> test was correct... >> >> Who is the "maintainer" of this class? >> >> So i guess 2 things need to be changed: >> >> BrFileCatalogManager::Register(): if (kInit) -> if (fStatus == kInit) >> BrFileCatalogInfo::Update(): change the "Select *" to only the >> nesecceary tables... >> maybe also remove the printf() statements in this file to Verbose or >> Debug... as I (a regular user) does not understand what the printing >> means... >> >> >> Cheers, >> Truls >> >> >> >> >>> hmmm. I looked into the guts of the BrFileCatalog manager... >>> >>> I guess I did it wrong. IT should be ::Register() before ::Init(). >>> >>> This should have prompted a warning in ::Register(), but it says: >>> >>> if (kInit) { >>> Warning(blablabla...) >>> return... >>> } >>> >>> It so happens that kInit is the first occurence in enum EDbStatus, >>> which I believe assigns kInit = 0! So the warning is never replayed... >>> >>> I guess is should read: >>> >>> if (fStatus == kInit) right??? >>> >>> Cheers, >>> Truls >>> >> >> >> On Thu, 2006-02-02 at 10:14 -0500, Hironori Ito wrote: >> >> >>> Ok. Looking more carefully (a few minutes), you >>> (BrFileCatalogInfo???) was doing the following query. >>> >>> "SELECT * FROM DataVersion WHERE ProductionVerId = 30 and RunNo = >>> 13899" >>> >>> Now, the content of this table (DataVersion) is the following. >>> +-----------------+-------------+------+-----+---------+----------------+ >>> >>> | Field | Type | Null | Key | Default | >>> Extra | >>> +-----------------+-------------+------+-----+---------+----------------+ >>> >>> | VersionId | int(11) | | PRI | NULL | >>> auto_increment | >>> | VersionNo | int(11) | | | 0 >>> | | >>> | DataTypeId | int(11) | | | 0 >>> | | >>> | InputVerId | int(11) | | | 0 >>> | | >>> | BratVer | varchar(15) | | | >>> | | >>> | RootVer | varchar(15) | | | >>> | | >>> | Script | text | | | >>> | | >>> | Creator | varchar(30) | | | >>> | | >>> | Comments | text | YES | | NULL >>> | | >>> | RunNo | int(11) | YES | | NULL >>> | | >>> | ProductionVerId | int(11) | | | 0 >>> | | >>> +-----------------+-------------+------+-----+---------+----------------+ >>> >>> >>> Now, from the above query, you don't need to get all data from this >>> table. Particularly, you don't need to get "Script, Creator, >>> Comments, BratVer, RootVer". Since you are tring to get all (by >>> Select * from DataVersion), this might be the reason for slowness. >>> After glacing at BrFileCatalogXXX codes, I did not see this >>> particular command. Someone who wrote this code can check this and >>> modify the code??? >>> >>> Hiro >>> >>> >>> Truls Martin Larsen wrote: >>> >>> >>> >>>> Hi, >>>> >>>> This is the code in banapp... It uses the BrFileCalalogManager, no >>>> custom query stuff. The idea is to give dst2tre a run range, begin run >>>> and end run number, and then the code retrives the location of the >>>> dsts >>>> that exist file catalog for given version and production, ie it just a >>>> loop from begin run number to end run number... >>>> >>>> So if I understand correctly: >>>> >>>> 1. Init FileCatalog manager. >>>> 2. Register runs >>>> 3. call filecatalog update >>>> 4 retive the filcatalog run. >>>> >>>> How can one do this i a proper way, without loading the DB to much??? >>>> Cheers, >>>> Truls >>>> >>>> On Thu, 2006-02-02 at 09:16 -0500, Hironori Ito wrote: >>>> >>>> >>>> >>>> >>>>> Hello, Truls. Are you querying FILECATALOG with some strange way >>>>> in your own program? MySQL server is having tough time querying >>>>> your request. If you are using your own query command, I suggest >>>>> you ( as well as any other person who use own SQL command for >>>>> query) to check if you can decrease the load to MySQL server. >>>>> Otherwise, the server response is too slow for other person. >>>>> >>>>> Hiro >>>>> >>>>> >>>>> >>>> >>> _______________________________________________ >>> Brahms-dev-l mailing list >>> Brahms-dev-l@lists.bnl.gov >>> http://lists.bnl.gov/mailman/listinfo/brahms-dev-l >>> >>> >> >> >> _______________________________________________ >> Brahms-dev-l mailing list >> Brahms-dev-l@lists.bnl.gov >> http://lists.bnl.gov/mailman/listinfo/brahms-dev-l >> >> >> _______________________________________________ Brahms-dev-l mailing list Brahms-dev-l@lists.bnl.gov http://lists.bnl.gov/mailman/listinfo/brahms-dev-lReceived on Thu Feb 2 12:25:09 2006
This archive was generated by hypermail 2.1.8 : Thu Feb 02 2006 - 12:25:17 EST