Pytables time performance -

-

i'm working on project related text detection in natural images. have train classifier , i'm using pytables store information. have:

  • 62 classes (a-z,a-z,0-9)
  • each class has between 100 , 600 tables
  • each table has 1 single column store 32bit float

  • each column has between 2^2 , 2^8 rows (depending on parameters)

    my problem after train classifier, takes lot of time read information in test. example: 1 database has 27900 tables (62 classes * 450 tables per class) , there 4 rows per table , took aprox 4hs read , retrieve information need. test program read each table 390 times (for classes a-z, a-z) , 150 times classes 0-9 info need. normal? tried use index option unique column , dont see performance. work on virtualmachine 2gb ram on hp pavillion dv6 (4gb ram ddr3, core2 duo).

this because column lookup on tables 1 of slower operations can , of information lives. have 2 basic options increase performance tables many columns , few rows:

  1. pivot structure such have table many rows , few columns.

  2. move more efficient data structure carray or earray every row / column.

additionally, can try using compression speed things up. sort of generic advice, because haven't included code.


Comments

Post a Comment

Popular posts from this blog

html5 - What is breaking my page when printing? -

-
Image
i have questioner , don't know why when printing it's creating break before first question. live demo (link) i able resolve issue removing fieldset container had wrapped in.. having fieldset element causing of fields grouped together, , since not fit on first page, being pushed second page. removing fieldset element, able firefox print first 2 questions on page 1 identically chrome. it seems not new problem firefox , fieldset not playing nice found old bugzilla ticket 2008 has persisted , still being commented on in 2014 (link) if retaining fieldset important, found solution else came here (link) <script type='text/javascript'> $(window).bind('beforeprint', function(){ $('fieldset').each( function(item) { $(this).replacewith($('<div class="fieldset">' + this.innerhtml + '</div>')); } ) }); $(window).bind(
Read more

c# - must be a non-abstract type with a public parameterless constructor in redis -

-
when save object, got below error must non-abstract type public parameterless constructor in order use parameter 't' in generic type or method 'servicestack.redis.redisclient.store<t>(t) redisclass.getinstnace().store(msg); // error here redisclass.getinstnace().save(); as third party's class, can not edit how save object? could create wrapper around third party object call constructor, store wrapper? e.g. public class mywrapper { public thirdpartyobject thirdpartyinstance { get; set; } public mywrapper() { thirdpartyinstance = new thirdpartyobject("constructors"); } }
Read more

ajax - PHP/JSON Login script (Twitter style) not setting sessions -

-
i'm having trouble getting twitter style login work properly, reason gets stuck at session_register("useremail"); session_register("userpass"); and doesn't redirect after it. if enter wrong account data, works , appropriate error message pops up. this full code: <?php if(empty($_post['email']) || empty($_post['password'])) { die(msg(0,"all fields required")); } if(!(preg_match("/^[\.a-z0-9_\-\+]+[@][a-z0-9_\-]+([.][a-z0-9_\-]+)+[a-z]{1,4}$/", $_post['email']))) die(msg(0,"you haven't provided valid e-mail")); $host="localhost"; $username="root"; $password="x"; $db_name="x"; $tbl_name="x"; mysql_connect("$host", "$username", "$password")or die("cannot connect"); mysql_select_db("$db_name")or die("cannot select db"); $useremail=$_post['email']; $userpass=
Read more