 |
|
|
TPIRick wrote:
Can anyone help with a proposed solution? How does Excel display very wide tables of data?
Have you ever had to find something in Excel when it's crammed with that amount of data? Once the user needs to scroll, he can also use a button to browse the next page.
You could virtualize the entire thing, drawing a grid and scrolling it's data - not the grid itself.
Bastard Programmer from Hell 
|
|
|
|
|
|
|
TPIRick wrote: Can anyone help with a proposed solution?
No one looks at that much data. Either they are looking for something in it or they want to manipulate it to do something else. First one means you ask the user what they want and only give them that. Second solution involves allowing the user to download the raw data (then they figure out how to manipulate it) or provide a way to manipulate it before presenting it to them.
|
|
|
|
|
|
|
Out of curiosity I performed a little experiment, with the following code inside the Form's Load handler (dgv is a DataGridView):
1 for (i=0; i<10000; i++) {
2 DataGridViewColumn col=new DataGridViewTextBoxColumn();
3 if (i==0) {
4 log("col.Width="+col.Width);
5 log("col.FillWeight="+col.FillWeight);
6 }
7 col.Name=i.ToString();
8 dgv.Columns.Add(col);
10 }
As is, it fails when i=655 with the Exception you've got, because the default FillWeight value is 100, and it somehow gets summed in an unsigned short which then overflows.
With line 8 uncommented, it runs fine. That is, if you have half an hour to spare, waiting for the DGV to get loaded with (empty) columns. After the long wait, the Form and the DGV appear, with column headers, scrollbars, and everything (except for rows and actual data, as I didn't provide any).
The conclusion can only be:
1. a DGV wasn't meant to hold that many columns;
2. your app needs another way to interact with your user.
|
|
|
|
|
|
|
Thought I would post my workaround in case it helps others.
Once the data is in a datatable I did a check on the number of columns and if it was very large then I disabled the datagridview to disable viewing. I then added code to allow the user to view the data via HTML. I created a humongo string which was an HTML representation of the data in the datatable. Nothing fancy just kept appending to the same StringBuilder object. Started with html cmds to set up the table then looped thru #rows, #columns spitting out tr, /tr, td, /td cmds with the cell contents in the middle. I then used a StreamWriter to write the humongo string to a file. I then added a button to allow the user to view the file that was created using the default browser. Button invocation calls "Process.Start(htmlFileCreated)".
I tried Firefox and IE 8/9 and was able to view the table even though it was 750 columns wide. Hope this helps someone.
Regards,
Rick
|
|
|
|
|
|
|
Hi all,
I have a scenario that is puzzling me and I'm guessing at the behavior and was wondering if anyone else knew what was happening under the skin of .Net.
I have a .Net 4.0 application on a clients' Windows Server 2008 64bit back end database is an ALPHA (not relevant).
The application is 1 form with literally hundreds of controls on it and also an image viewer. It's takes up quite a bit of screen space, and has multiple tabs (at least 8) to hold all the controls.
The puzzling scenario is this:
1. Open up a simple list
2. Select an item from the list to open up the form as described above - this takes over 1min 15secs
2. Close the form (return to list in point 1)
3. Open up the same form from the list loading the controls in exactly the same way with the same queries behind it (assuming not relevant from the logs I've looked at), and it can open in 10 seconds flat.
From the looks of the memory print in Task Manager, the memory obviously shoots up when the form is first opened, but doesn't go down too far after closing the form. On reopening the form again, the memory doesn't go up much but only takes 10 seconds to open.
I'm thinking that .Net is loading up a load of stuff (!for want of a better word!) into memory on first load of form, and subsequent forms use what has previously been loaded into memory.
If it's not this that is happening, I'm not sure what is happening, as I have been asked by our client to try and make the first load of the form as quick as each subsequent one, and not sure if this is indeed possible.
Any light shed on this would be most appreciated.
Many thanks
Julian
|
|
|
|
|
|
|
My initial thought here, as you haven't said what database you are hitting, is that the database connection is being pooled. On the first invocation, the connection is being created which is a time consuming process. Further invocations are able to take the pooled database connection.
You are also correct that the .NET runtime will also grab extra memory which is not released immediately, allowing space for the application to load the form in again.
|
|
|
|
|
|
|
Hi
The connection to the database is active before the form loads, and is not dropped for the duration of the application lifetime (it's a secure self contained network).
The database as stated is an Alpha database running on a VMS server. We have a windows dll interface to it.
I'm interested in your point about .Net not releasing memory. If I create a new Form each time (Form1 creating Form2 - Form2 being the Form in question), do you think that .Net will intelligently store stuff it needs and call on it again when I create a new Form2?
Thanks for your response
Julian
|
|
|
|
|
|
|
Assuming your description of the situation is somewhat accurate, I can only conclude one did a few things wrongly in the implementation. Adding some simple logging (with time stamps obviously) to the app would allow to pinpoint the major problems. And yes I expect there to be more than one. To name some candidates:
- user authorization on the network
- user authorization on the DB
- establishing the first DB connection
- building the GUI
- obtaining all the data
- stuffing the data in the GUI.
You'll need some facts before you can effectively tackle this, so start logging!
Alternatively, and maybe harder to implement and interpret, try simpler set-ups.
BTW: your app may or may not contain extra implementation features to make subsequent operations faster, think caching. The more of these you have, the more impossible it should become for the first operation to be as fast as subsequent ones.
FWIW: In Windows, dialog windows can be hidden and shown over and over, without recreating them. So even the most complex dialog could be shown again in a fraction of a second (and anyone can come up with ways to make things slower).
PS: if you have tab pages on a Form, then there is no need to fully populate them all before showing anything to the user, as he can see only one of them at a time. The right approach would be to either load the others in a lazy way, i.e. only when needed; or in a background operation, i.e. after the essential work has finished.
|
|
|
|
|
|
|
Hi
Yep, already logging like crazy. I have 2 logs on the go,
Log1: almost every function logs it's 'In' and 'Out' times in milliseconds
Log2: Every single query performed on the database is logged to a separate file and time stamped.
I can see from the combination of these what takes time.
So I can see for instance that the Form Shown event finishes at time x (on first showing), and the form doesn't actually appear on the screen until 30 seconds later. Also, no queries occur or other functions called during that 30 seconds.
The database connection (as stated, an Alpha database running on a VMS server) is never dropped, it is continually open and open before the form is first constructed.
Untangling the data from the tabs would be too time consuming (showing data as needed), and we would be issuing queries multiple times, when it would only need to be once at the beginning. However I have taken this approach with the second tab control (there are 2 on the form - 1 with 1 image per tab, and 1 with controls on each tab). The image tab control does indeed load it's image only when necessary as image handling can take time.
I think it's too late to re-show the dialog and simply re-populate the controls on showing the second time, and the second time isn't a problem anyway.
I'm also using threads to help populate the form, attempting as best as possible to do many things at one time.
So, still not sure.
Julian
|
|
|
|
|
|
|
julian@giant wrote: he form doesn't actually appear on the screen until 30 seconds later.
That is unacceptable, and I've never seen such thing.
julian@giant wrote: The database connection (as stated, an Alpha database running on a VMS server) is never dropped
That sounds wrong. You should not keep connections open, .NET has connection pooling, meaning you open-query-close (actually use the using construct for getting a connection), and .NET will silently cache and reuse connections for you. With only one connection, you are preventing efficient multi-threading to the DB.
julian@giant wrote: I'm also using threads to help populate the form
I know of many ways to get that wrong. Too many thread switches would be one; deadlocks another. Are you using Control.InvokeRequired/Control.Invoke? Are you overusing them for each little piece of displayable information? And please don't tell me you have touched Control.CheckForIllegalCrossThreadCalls!
Suggestion: use Task Manager and watch your client's CPU load; if it is near zero for several seconds, something is very wrong (deadlock suspected); if it is near 1/N (with N cores), a busy loop may be the culprit. Also have a look at your server's CPU load (assuming you can get your client to be the only client). Can you correlate both load curves? Any dead periods overall?
PS: no, .NET isn't caching anything in general, when it does, it is documented. DB connections is an example. Windows does cache files. And .NET is not throwing away things as long as it doesn't need the memory for something else. However a new Form would be independent of a previous instance of the same Form (except for its code to be loaded and JIT-ted just once).
|
|
|
|
|
|
|
Hi Luc
Thanks.
Yes I agree, 30 seconds is totally unacceptable! And strange. I'm trying to find out what's causing it now. On the face of it, the program is not doing anything! But I do see the server memory going up and up during that 30 seconds.
The database connection thing has been the way the company does things for along time. I've tried a separate app that does indeed do what you're suggesting (putting queries in threads etc.), but somehow, the Alpha database (it came out of the Ark) can't handle queries being done on threads other than the main thread. So, I've stuck with one connection.
I'm pretty sure there's no threading issue during that 30 seconds, but it's something i will check.
Thanks again for your response.
Julian
|
|
|
|
|
|
|
if that 30 seconds is wasted on the server's side (and memory is spent more than expected?), it tells me either your query is awful, or your DB is performing some silly tasks such as say "dynamic indexing". Check your query, and the indexes that are in place.
But then you said the 30 seconds elapsed between the end of Shown and the Form appearing? There should be nothing there, in fact a Form normally is visible when the Shown handler begins.
I'm not convinced yet all your observations are correct (I learned long ago to suspect everything while debugging and/or hunting for performance).
|
|
|
|
|
|
|
Thanks Luc,
You've helped me identify some places to look and I shall double check the Form Show event.
Don't think that the database has anything to do with it, we issue a query and it comes back, (some take longer than others) but I've never experienced a query to the DB causing a hang of sorts after it has returned. That's not to say it isn't a first. The DB as I say is old and the interface to it seems to be not overly supported.
Thinking about it, that is an area i haven't looked into. Time to investigate the Alpha logs.. oh no... save me....
Cheers
Julian
|
|
|
|
|
|
|
OK, have now got this sorted.
Previously the .exe I was doing performance enhancements on was 450Mb in size and took 1m30s to load each form it needed to create (from within the app itself)!
I managed to get this down to 1m15s on the first time the form was opened and 10 seconds every subsequent time it was opened from within the app. Memory usage was down to 150Mb. I was quite pleased with myself.
The target application server is 64bit. Setting the target CPU in the project settings of the application to x86 from x64 brought the size of the .exe down to around 60Mb and there is no delay on the initial creation of the first form from within the app.
Knew I'd get there in the end.
Thanks for all your input into this guys. Some most interesting suggestions.
Cheers
Julian
|
|
|
|
|
|
|
Thanks for the update.
I have no idea why a target change would have such drastic effects. Do you? Or what haven't you told us??
|
|
|
|
|
|
|
I don't know what I haven't told you .... yet....
|
|
|
|
|
|
|
Luc Pattyn wrote:
he form doesn't actually appear on the screen until 30
seconds later.
That is unacceptable, and I've never seen such
thing.
I have. When there's hundred of controls on the form. Creating a control, and the underying window, is expensive. Then, if he's populating list controls with hundreds of items, that isn't exactly setting an speed records either.
|
|
|
|
|
|
|
Thanks Dave. I am indeed populating many DataGridViews, which at this point should have been loaded with their data. Not hundreds of rows per DGV tho.
I'll bear it in mind.
Cheers
Julian
|
|
|
|
|
|
|
I agree having lots of Controls isn't a great idea, however I do expect Shown to be true to its name, especially as the Controls should all be created by the main thread which is also where the Shown handler is running. So I'd expect a complete Paint to have finished before Shown fires. You're saying that is not guaranteed?
[ADDED] I just performed a little experiment, with a Form's Load handler adding 5000 labels. The Load handler took over 20 seconds, the Shown handler sounded its programmed beep when the Form became visible. And the first Paint event came right after the Shown event, not before! So Windows claims a Form is shown without it being painted??
[/ADDED]
|
|
|
|
|
|
|
Thanks Luc. I'll give that a go myself but spark off the form from another form, and try it a second time and see if it takes just as long on the second time.
Interesting.
Cheers
Julian
modified 17 Feb '12.
|
|
|
|
|
|
|
I just tried the same, with similar results. I added 5000 labels to the form, each with a different x/y position. Took about 20 seconds before they displayed. Then I tried resizing the form and I could actually see the labels repainting over several seconds. The problem in this case seems to be a very inefficient repaint algorithm.
Probably something like an O(N^2) inefficiency. I've seen this before when I implemented a repaint algorithm for a 2D game that found the top rectangle, painted it, then recursively descended down the 8 surrounding quadrants to check the remaining rectangles. With enough rectangles, it becomes more efficient to just draw all of the rectangles in their entirety, from bottom to top. I think the game Sonic had this problem too... if you get a bunch of rings on the screen at the same time, the animation will slow down.
Pretty amazing that such an inefficiency exists in Windows Forms.
|
|
|
|
|
|
|
AspDotNetDev wrote: The problem in this case seems to be a very inefficient repaint algorithm.
Lot's of handle's means lot's of communication. Easily remedied; stop using that ridiculous amount of controls.
Bastard Programmer from Hell
|
|
|
|
|
|
|
Um... thanks for that Eddy, but what if your Dialog requires lots of controls? J
|
|
|
|
|
|
|
Lots of controls means lots of message-pumps. Now, instead of creating 300 checkboxes and stacking them, one could draw 300 checkboxes on a single control. Then you catch the mouseclicks on your panel, and check/uncheck that checkbox manually, and update the drawing. That eliminates 299 message pumps, and some handles.
Bastard Programmer from Hell
|
|
|
|
|
|
|
If I were you, I'd be looking into using an application profiler to see where the time is being taken up. It will make your life a lot easier because it will enable you to see what your application is really doing - look particularly for excessively long calls for few calls, or operations with excessive number of calls.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Rant 
Admin